MonetDB: Jun2020 - Fixes changelog syntax.

[Sjoerd Mullender]

Changeset: 539aa9553c9e for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=539aa9553c9e
Modified Files:
MonetDB.spec
debian/changelog
Branch: Jun2020
Log Message:

Fixes changelog syntax.


diffs (67 lines):

diff --git a/MonetDB.spec b/MonetDB.spec
--- a/MonetDB.spec
+++ b/MonetDB.spec
@@ -5062,20 +5062,20 @@ fi
 
 * Fri May 13 2011 Sjoerd Mullender  - 11.3.3-20110517
 - gdk: Fixed a bug where large files (> 2GB) didn't always get deleted on
-Windows.
+  Windows.
 
 * Wed May 11 2011 Fabian Groffen  - 11.3.3-20110517
 - java: Insertion via PreparedStatement and retrieval via ResultSet of 
timestamp
-and time fields with and without timezones was improved to better
-respect timezones, as partly indicated in bug #2781.
+  and time fields with and without timezones was improved to better
+  respect timezones, as partly indicated in bug #2781.
 
 * Wed May 11 2011 Sjoerd Mullender  - 11.3.3-20110517
 - monetdb5: Fixed a bug in conversion from string to the URL type.  The bug was
-an incorrect call to free().
+  an incorrect call to free().
 
 * Wed Apr 27 2011 Sjoerd Mullender  - 11.3.3-20110517
 - geom: Fixed various problems so that now all our tests work correctly on
-all our testing platforms.
+  all our testing platforms.
 
 * Thu Apr 21 2011 Sjoerd Mullender  - 11.3.1-20110421
 - Rebuilt.
diff --git a/debian/changelog b/debian/changelog
--- a/debian/changelog
+++ b/debian/changelog
@@ -5846,29 +5846,29 @@ monetdb (11.3.3) unstable; urgency=low
 monetdb (11.3.3) unstable; urgency=low
 
   * gdk: Fixed a bug where large files (> 2GB) didn't always get deleted on
-Windows.
+Windows.
 
  -- Sjoerd Mullender   Fri, 13 May 2011 16:45:38 +0200
 
 monetdb (11.3.3) unstable; urgency=low
 
   * java: Insertion via PreparedStatement and retrieval via ResultSet of 
timestamp
-and time fields with and without timezones was improved to better
-respect timezones, as partly indicated in bug #2781.
+and time fields with and without timezones was improved to better
+respect timezones, as partly indicated in bug #2781.
 
  -- Fabian Groffen   Wed, 11 May 2011 16:45:38 +0200
 
 monetdb (11.3.3) unstable; urgency=low
 
   * monetdb5: Fixed a bug in conversion from string to the URL type.  The bug 
was
-an incorrect call to free().
+an incorrect call to free().
 
  -- Sjoerd Mullender   Wed, 11 May 2011 16:45:38 +0200
 
 monetdb (11.3.3) unstable; urgency=low
 
   * geom: Fixed various problems so that now all our tests work correctly on
-all our testing platforms.
+all our testing platforms.
 
  -- Sjoerd Mullender   Wed, 27 Apr 2011 16:45:38 +0200
 
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Jun2020 - New subquery crash :( The any/all rewriter, s...

[Pedro Ferreira]

Changeset: dd18fd8c495f for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=dd18fd8c495f
Modified Files:
sql/test/subquery/Tests/subquery5.sql
Branch: Jun2020
Log Message:

New subquery crash :( The any/all rewriter, still doesn't take into account 
some type of joins


diffs (24 lines):

diff --git a/sql/test/subquery/Tests/subquery5.sql 
b/sql/test/subquery/Tests/subquery5.sql
--- a/sql/test/subquery/Tests/subquery5.sql
+++ b/sql/test/subquery/Tests/subquery5.sql
@@ -139,6 +139,20 @@ SELECT (VALUES(col1, col2)) FROM another
 
 SELECT (VALUES(col1), (col2)) FROM another_t; --error, more than one row 
returned by a subquery used as an expression
 
+SELECT * FROM integers i1 LEFT OUTER JOIN integers i2 ON i2.i = ANY(SELECT 
SUM(i2.i + i3.i) FROM integers i3) = NOT EXISTS(SELECT MIN(i1.i) OVER ());
+   -- 1 3
+   -- 1 2
+   -- 1 1
+   -- 2 3
+   -- 2 2
+   -- 2 1
+   -- 3 3
+   -- 3 2
+   -- 3 1
+   -- NULL 3
+   -- NULL 2
+   -- NULL 1
+
 DROP FUNCTION evilfunction(INT);
 DROP TABLE tbl_ProductSales;
 DROP TABLE another_T;
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: scoping - Use bool and prepand hidden MUL frame with %,...

[Pedro Ferreira]

Changeset: 692e5706a7d3 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=692e5706a7d3
Modified Files:
sql/server/rel_semantic.c
sql/server/rel_sequence.c
sql/server/rel_updates.c
sql/server/sql_mvc.c
sql/server/sql_mvc.h
Branch: scoping
Log Message:

Use bool and prepand hidden MUL frame with %, so it cannot be never accidentaly 
intercepted by an user


diffs (63 lines):

diff --git a/sql/server/rel_semantic.c b/sql/server/rel_semantic.c
--- a/sql/server/rel_semantic.c
+++ b/sql/server/rel_semantic.c
@@ -200,7 +200,7 @@ rel_semantic(sql_query *query, symbol *s
dnode *d;
sql_rel *r = NULL;
 
-   if (!stack_push_frame(sql, "MUL"))
+   if (!stack_push_frame(sql, "%%MUL"))
return sql_error(sql, 02, SQLSTATE(HY013) 
MAL_MALLOC_FAIL);
for (d = s->data.lval->h; d; d = d->next) {
symbol *sym = d->data.sym;
diff --git a/sql/server/rel_sequence.c b/sql/server/rel_sequence.c
--- a/sql/server/rel_sequence.c
+++ b/sql/server/rel_sequence.c
@@ -112,7 +112,7 @@ rel_create_seq(
seq->bedropped = bedropped;
res = rel_seq(sql->sa, ddl_create_seq, s->base.name, seq, NULL, NULL);
/* for multi statements we keep the sequence around */
-   if (res && stack_has_frame(sql, "MUL") != 0) {
+   if (res && stack_has_frame(sql, "%%MUL") != 0) {
if (!stack_push_rel_view(sql, name, rel_dup(res)))
return sql_error(sql, 02, SQLSTATE(HY013) 
MAL_MALLOC_FAIL);
}
diff --git a/sql/server/rel_updates.c b/sql/server/rel_updates.c
--- a/sql/server/rel_updates.c
+++ b/sql/server/rel_updates.c
@@ -2071,9 +2071,9 @@ rel_updates(sql_query *query, symbol *s)
 {
mvc *sql = query->sql;
sql_rel *ret = NULL;
-   int old = sql->use_views;
+   bool old = sql->use_views;
 
-   sql->use_views = 1;
+   sql->use_views = true;
switch (s->token) {
case SQL_COPYFROM:
{
diff --git a/sql/server/sql_mvc.c b/sql/server/sql_mvc.c
--- a/sql/server/sql_mvc.c
+++ b/sql/server/sql_mvc.c
@@ -829,7 +829,7 @@ mvc_create(int clientid, backend_stack s
m->topframes = 0;
m->frame = 0;
 
-   m->use_views = 0;
+   m->use_views = false;
m->argmax = MAXPARAMS;
m->args = NEW_ARRAY(atom*, m->argmax);
if (!m->frames || !m->args) {
diff --git a/sql/server/sql_mvc.h b/sql/server/sql_mvc.h
--- a/sql/server/sql_mvc.h
+++ b/sql/server/sql_mvc.h
@@ -122,7 +122,7 @@ typedef struct mvc {
int topframes;
int sizeframes;
int frame;
-   int use_views;
+   bool use_views;
atom **args;
int argc;
int argmax;
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Jun2020 - Defensive exp_name calls before exp_ref

[Pedro Ferreira]

Changeset: dd16a2589e0f for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=dd16a2589e0f
Modified Files:
sql/server/rel_select.c
sql/server/rel_updates.c
Branch: Jun2020
Log Message:

Defensive exp_name calls before exp_ref


diffs (24 lines):

diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c
--- a/sql/server/rel_select.c
+++ b/sql/server/rel_select.c
@@ -4414,6 +4414,8 @@ rel_order_by(sql_query *query, sql_rel *
if (!found) {
append(rel->exps, e);
} else {
+   if (!exp_name(found))
+   exp_label(sql->sa, 
found, ++sql->label);
e = found;
}
e = exp_ref(sql->sa, e);
diff --git a/sql/server/rel_updates.c b/sql/server/rel_updates.c
--- a/sql/server/rel_updates.c
+++ b/sql/server/rel_updates.c
@@ -776,6 +776,8 @@ rel_update_join_idx(mvc *sql, const char
 
/* FOR MATCH FULL/SIMPLE/PARTIAL see above */
/* Currently only the default MATCH SIMPLE is supported */
+   if (!exp_name(upd))
+   exp_label(sql->sa, upd, ++sql->label);
upd = exp_ref(sql->sa, upd);
lnl = exp_unop(sql->sa, upd, isnil);
set_has_no_nil(lnl);
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: scoping - Approved output

[Pedro Ferreira]

Changeset: c7d206c63587 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=c7d206c63587
Added Files:
sql/test/scoping/Tests/predefinedvariable.stable.err
sql/test/scoping/Tests/predefinedvariable.stable.out
Modified Files:
sql/test/scoping/Tests/All
sql/test/scoping/Tests/predefinedvariable.sql
Branch: scoping
Log Message:

Approved output


diffs (157 lines):

diff --git a/sql/test/scoping/Tests/All b/sql/test/scoping/Tests/All
--- a/sql/test/scoping/Tests/All
+++ b/sql/test/scoping/Tests/All
@@ -4,4 +4,5 @@ HAVE_PYMONETDB?scoping03
 
 schema_disambiguity
 ambiguity
+predefinedvariable
 simple_nested
diff --git a/sql/test/scoping/Tests/predefinedvariable.sql 
b/sql/test/scoping/Tests/predefinedvariable.sql
--- a/sql/test/scoping/Tests/predefinedvariable.sql
+++ b/sql/test/scoping/Tests/predefinedvariable.sql
@@ -9,6 +9,8 @@ select sys.optimizer;   -- can we find it 
 select tmp.optimizer;  -- should not be found
 
 -- the optimizer variable can be reassigned a value
+declare aux string;
+set aux = (select sys.optimizer);
 set optimizer = 'minimal_pipe';
 select optimizer;
 
@@ -43,12 +45,11 @@ create procedure poo2()
 begin
set sys.optimizer='volcano_pipe';   
 end;
--- returns: syntax error, unexpected END in: "end"
-
 
 create procedure poo4()
 begin
set optimizer='deep-pipe';
-   select optimizer;
+   select optimizer; --error, regular select statements not allowed inside 
procedures (disallowed by the parser)
 end;
--- returns : syntax error, unexpected END in: "end"
+
+set optimizer = (select sys.aux);
diff --git a/sql/test/scoping/Tests/predefinedvariable.stable.err 
b/sql/test/scoping/Tests/predefinedvariable.stable.err
new file mode 100644
--- /dev/null
+++ b/sql/test/scoping/Tests/predefinedvariable.stable.err
@@ -0,0 +1,30 @@
+stderr of test 'predefinedvariable` in directory 'sql/test/scoping` itself:
+
+
+# 14:57:50 >  
+# 14:57:50 >  "mclient" "-lsql" "-ftest" "-tnone" "-Eutf-8" "-i" "-e" 
"--host=/var/tmp/mtest-467788" "--port=39099"
+# 14:57:50 >  
+
+MAPI  = (monetdb) /var/tmp/mtest-467788/.s.monetdb.39099
+QUERY = select tmp.optimizer;  -- should not be found
+ERROR = !SELECT: no such column 'tmp.optimizer'
+CODE  = 42S22
+MAPI  = (monetdb) /var/tmp/mtest-467788/.s.monetdb.39099
+QUERY = create procedure poo4()
+begin
+   set optimizer='deep-pipe';
+   select optimizer; --error, regular select statements not 
allowed inside procedures (disallowed by the parser)
+ERROR = !syntax error, unexpected SCOLON, expecting INTO in: "create procedure 
poo4()
+!begin
+!  set optimizer='deep-pipe';
+!  select optimizer;"
+CODE  = 42000
+MAPI  = (monetdb) /var/tmp/mtest-467788/.s.monetdb.39099
+QUERY = end;
+ERROR = !syntax error, unexpected END in: "end"
+CODE  = 42000
+
+# 14:57:50 >  
+# 14:57:50 >  "Done."
+# 14:57:50 >  
+
diff --git a/sql/test/scoping/Tests/predefinedvariable.stable.out 
b/sql/test/scoping/Tests/predefinedvariable.stable.out
new file mode 100644
--- /dev/null
+++ b/sql/test/scoping/Tests/predefinedvariable.stable.out
@@ -0,0 +1,80 @@
+stdout of test 'predefinedvariable` in directory 'sql/test/scoping` itself:
+
+
+# 14:57:50 >  
+# 14:57:50 >  "mclient" "-lsql" "-ftest" "-tnone" "-Eutf-8" "-i" "-e" 
"--host=/var/tmp/mtest-467788" "--port=39099"
+# 14:57:50 >  
+
+#select optimizer;
+% . # table_name
+% single_value # name
+% varchar # type
+% 12 # length
+[ "default_pipe"   ]
+#select current_schema;
+% . # table_name
+% single_value # name
+% varchar # type
+% 3 # length
+[ "sys"]
+#select sys.optimizer; -- can we find it there?
+% . # table_name
+% single_value # name
+% varchar # type
+% 12 # length
+[ "default_pipe"   ]
+#declare aux string;
+#set aux = (select sys.optimizer);
+#set optimizer = 'minimal_pipe';
+#select optimizer;
+% . # table_name
+% single_value # name
+% varchar # type
+% 12 # length
+[ "minimal_pipe"   ]
+#set sys.optimizer = 'minimal_pipe';
+#select sys.optimizer;
+% . # table_name
+% single_value # name
+% varchar # type
+% 12 # length
+[ "minimal_pipe"   ]
+#create table mynewone( i integer, optimizer integer);
+#insert into mynewone(i, optimizer) values(1,2);   -- to marked as 
sheelding outer definition (ako Pythonic)
+[ 1]
+#select i, optimizer from mynewone;-- 
ambiguous
+% sys.mynewone,sys.mynewone # table_name
+% i,   optimizer # name
+% int, int # type
+% 1,   1 # length
+[ 1,   2   ]
+#select i, sys.optimizer from mynewone;-- 
should be recognized
+% sys.mynewone,. # table_name
+% i,   single_value # name
+% int, varchar # type
+% 1,   12 # length
+[ 1,   "minimal_pipe"  ]
+#create function foo()
+#returns integer
+#begin
+#  return optimizer;
+#end;
+#create function foo2()
+#returns integer
+#begin
+#  return sys.optimizer;
+#end;
+#create procedure poo()
+#begin
+#

MonetDB: Jun2020 - Backported changeset 46d667c9bffd into Jun2020

[Pedro Ferreira]

Changeset: 8687fe5adb6e for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8687fe5adb6e
Modified Files:
monetdb5/optimizer/opt_support.c
Branch: Jun2020
Log Message:

Backported changeset 46d667c9bffd into Jun2020


diffs (13 lines):

diff --git a/monetdb5/optimizer/opt_support.c b/monetdb5/optimizer/opt_support.c
--- a/monetdb5/optimizer/opt_support.c
+++ b/monetdb5/optimizer/opt_support.c
@@ -302,7 +302,8 @@ isUpdateInstruction(InstrPtr p){
getFunctionId(p) == updateRef ||
getFunctionId(p) == deleteRef ||
getFunctionId(p) == growRef ||
-   getFunctionId(p) == clear_tableRef))
+   getFunctionId(p) == clear_tableRef ||
+   getFunctionId(p) == setVariableRef))
return TRUE;
if ( getModuleId(p) == batRef &&
   ( getFunctionId(p) == appendRef ||
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: scoping - Merged with default

[Pedro Ferreira]

Changeset: fc86c78da595 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fc86c78da595
Modified Files:
gdk/gdk.h
gdk/gdk_bat.c
gdk/gdk_bbp.c
gdk/gdk_hash.c
gdk/gdk_imprints.c
gdk/gdk_orderidx.c
gdk/gdk_storage.c
gdk/gdk_system.c
gdk/gdk_system.h
gdk/gdk_utils.c
monetdb5/mal/mal_client.c
monetdb5/mal/mal_dataflow.c
monetdb5/modules/mal/mal_mapi.c
monetdb5/modules/mal/tablet.c
sql/backends/monet5/mal_backend.c
sql/backends/monet5/sql_gencode.c
sql/backends/monet5/sql_statement.c
sql/common/sql_types.c
sql/server/rel_exp.c
sql/server/rel_select.c
sql/server/rel_unnest.c
sql/server/sql_atom.c
sql/server/sql_partition.c
sql/server/sql_query.c
sql/server/sql_semantic.c
sql/storage/bat/bat_storage.c
sql/storage/sql_storage.h
sql/storage/store.c
sql/test/Tests/hot-snapshot.py
sql/test/miscellaneous/Tests/simple_selects.sql
sql/test/miscellaneous/Tests/simple_selects.stable.err
sql/test/subquery/Tests/subquery5.sql
sql/test/subquery/Tests/subquery5.stable.err
sql/test/subquery/Tests/subquery5.stable.out
Branch: scoping
Log Message:

Merged with default


diffs (truncated from 879 to 300 lines):

diff --git a/gdk/gdk.h b/gdk/gdk.h
--- a/gdk/gdk.h
+++ b/gdk/gdk.h
@@ -1709,7 +1709,7 @@ typedef struct threadStruct {
 * into this array + 1 (0 is
 * invalid) */
ATOMIC_TYPE pid;/* thread id, 0 = unallocated */
-   char name[16];
+   char name[MT_NAME_LEN];
void *data[THREADDATA];
uintptr_t sp;
 } *Thread;
diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c
--- a/gdk/gdk_bat.c
+++ b/gdk/gdk_bat.c
@@ -121,7 +121,7 @@ BATcreatedesc(oid hseq, int tt, bool hea
bn->tvheap->parentid = bn->batCacheid;
bn->tvheap->farmid = BBPselectfarm(role, bn->ttype, varheap);
}
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "BATlock%d", bn->batCacheid); /* fits */
MT_lock_init(>batIdxLock, name);
bn->batDirtydesc = true;
diff --git a/gdk/gdk_bbp.c b/gdk/gdk_bbp.c
--- a/gdk/gdk_bbp.c
+++ b/gdk/gdk_bbp.c
@@ -910,7 +910,7 @@ BBPreadEntries(FILE *fp, unsigned bbpver
bn->batCount = (BUN) count;
bn->batInserted = bn->batCount;
bn->batCapacity = (BUN) capacity;
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "BATlock%d", bn->batCacheid); /* 
fits */
MT_lock_init(>batIdxLock, name);
 
diff --git a/gdk/gdk_hash.c b/gdk/gdk_hash.c
--- a/gdk/gdk_hash.c
+++ b/gdk/gdk_hash.c
@@ -957,7 +957,7 @@ BAThash(BAT *b)
if (BBP_status(b->batCacheid) & BBPEXISTING && !b->theap.dirty 
&& !GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "hashsync%d", 
b->batCacheid);
MT_lock_unset(>batIdxLock);
if (MT_create_thread(, BAThashsync, b,
diff --git a/gdk/gdk_imprints.c b/gdk/gdk_imprints.c
--- a/gdk/gdk_imprints.c
+++ b/gdk/gdk_imprints.c
@@ -534,7 +534,7 @@ BATimprints(BAT *b)
!GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "impssync%d", 
b->batCacheid);
if (MT_create_thread(, BATimpsync, b,
 MT_THR_DETACHED, name) < 0)
diff --git a/gdk/gdk_orderidx.c b/gdk/gdk_orderidx.c
--- a/gdk/gdk_orderidx.c
+++ b/gdk/gdk_orderidx.c
@@ -169,7 +169,7 @@ persistOIDX(BAT *b)
!GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "oidxsync%d", b->batCacheid);
if (MT_create_thread(, BATidxsync, b,
 MT_THR_DETACHED, name) < 0)
@@ -479,7 +479,7 @@ GDKmergeidx(BAT *b, BAT**a, int n_ar)
b->batInserted == b->batCount) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "oidxsync%d", b->batCacheid);
if (MT_create_thread(, BATidxsync, b,
 MT_THR_DETACHED, name) < 0)
diff --git a/gdk/gdk_storage.c b/gdk/gdk_storage.c
--- a/gdk/gdk_storage.c
+++ b/gdk/gdk_storage.c

MonetDB: scoping - Approved output

[Pedro Ferreira]

Changeset: c2dedeb9eec1 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=c2dedeb9eec1
Modified Files:
clients/Tests/MAL-signatures.stable.out
clients/Tests/MAL-signatures.stable.out.int128
Branch: scoping
Log Message:

Approved output


diffs (42 lines):

diff --git a/clients/Tests/MAL-signatures.stable.out 
b/clients/Tests/MAL-signatures.stable.out
--- a/clients/Tests/MAL-signatures.stable.out
+++ b/clients/Tests/MAL-signatures.stable.out
@@ -14112,7 +14112,7 @@ stdout of test 'MAL-signatures` in direc
 [ "sql",   "export_table", "unsafe pattern sql.export_table(fname:str, 
fmt:str, colsep:str, recsep:str, qout:str, nullrep:str, onclient:int, 
tbl:bat[:str], attr:bat[:str], tpe:bat[:str], len:bat[:int], scale:bat[:int], 
cols:bat[:any]...):int ",   "mvc_export_table_wrap;",   "Prepare a 
table result set for the COPY INTO stream"   ]
 [ "sql",   "first_value",  "pattern sql.first_value(b:any_1, s:lng, 
e:lng):any_1 ","SQLfirst_value;",  "return the first value of 
groups"  ]
 [ "sql",   "flush_log","command sql.flush_log():void ",
"SQLflush_log;","start flushing the write ahead log"]
-[ "sql",   "getVariable",  "pattern sql.getVariable(mvc:int, 
varname:str):any_1 ", "getVariable;", "Get the value of a session variable"   ]
+[ "sql",   "getVariable",  "pattern sql.getVariable(mvc:int, sname:str, 
varname:str):any_1 ",  "getVariable;", "Get the value of a session 
variable"   ]
 [ "sql",   "getVersion",   "command sql.getVersion(clientid:int):lng ",
"mvc_getVersion;",  "Return the database version identifier for a client."  
]
 [ "sql",   "get_value","pattern sql.get_value(sname:str, 
sequence:str):lng ",  "mvc_get_value;",   "return the current value of the 
sequence"  ]
 [ "sql",   "grow", "pattern sql.grow(tid:bat[:oid], X_0:any_1):int ",  
"mvc_grow_wrap;",   "Resize the tid column of a declared table."]
@@ -14178,7 +14178,7 @@ stdout of test 'MAL-signatures` in direc
 [ "sql",   "row_number",   "pattern sql.row_number(b:any_1, p:bit, 
o:bit):int ",   "SQLrow_number;",   "return the row_numer-ed groups"
]
 [ "sql",   "rt_credentials",   "pattern 
sql.rt_credentials(tablename:str) (uri:bat[:str], username:bat[:str], 
hash:bat[:str]) ",   "sql_rt_credentials_wrap;", "Return the remote 
table credentials for the given table"   ]
 [ "sql",   "sessions", "pattern sql.sessions() (id:bat[:int], 
user:bat[:str], start:bat[:timestamp], idle:bat[:timestamp], 
optmizer:bat[:str], stimeout:bat[:int], qtimeout:bat[:int], wlimit:bat[:int], 
mlimit:bat[:int]) ",  "sql_sessions_wrap;",   "SQL export table of active 
sessions, their timeouts and idle status"   ]
-[ "sql",   "setVariable",  "pattern sql.setVariable(mvc:int, varname:str, 
value:any_1):int ",  "setVariable;", "Set the value of a session variable"  
 ]
+[ "sql",   "setVariable",  "pattern sql.setVariable(mvc:int, sname:str, 
varname:str, value:any_1):int ",   "setVariable;", "Set the value of a session 
variable"   ]
 [ "sql",   "shrink",   "pattern sql.shrink(sch:str, tbl:str):void ",   
"SQLshrink;",   "Consolidate the deletion table over all columns using 
shrinking"   ]
 [ "sql",   "shutdown", "pattern sql.shutdown(delay:bte):str ", 
"SQLshutdown_wrap;",""  ]
 [ "sql",   "shutdown", "pattern sql.shutdown(delay:bte, force:bit):str 
",  "SQLshutdown_wrap;",""  ]
diff --git a/clients/Tests/MAL-signatures.stable.out.int128 
b/clients/Tests/MAL-signatures.stable.out.int128
--- a/clients/Tests/MAL-signatures.stable.out.int128
+++ b/clients/Tests/MAL-signatures.stable.out.int128
@@ -19569,7 +19569,7 @@ stdout of test 'MAL-signatures` in direc
 [ "sql",   "export_table", "unsafe pattern sql.export_table(fname:str, 
fmt:str, colsep:str, recsep:str, qout:str, nullrep:str, onclient:int, 
tbl:bat[:str], attr:bat[:str], tpe:bat[:str], len:bat[:int], scale:bat[:int], 
cols:bat[:any]...):int ",   "mvc_export_table_wrap;",   "Prepare a 
table result set for the COPY INTO stream"   ]
 [ "sql",   "first_value",  "pattern sql.first_value(b:any_1, s:lng, 
e:lng):any_1 ","SQLfirst_value;",  "return the first value of 
groups"  ]
 [ "sql",   "flush_log","command sql.flush_log():void ",
"SQLflush_log;","start flushing the write ahead log"]
-[ "sql",   "getVariable",  "pattern sql.getVariable(mvc:int, 
varname:str):any_1 ", "getVariable;", "Get the value of a session variable"   ]
+[ "sql",   "getVariable",  "pattern sql.getVariable(mvc:int, sname:str, 
varname:str):any_1 ",  "getVariable;", "Get the value of a session 
variable"   ]
 [ "sql",   "getVersion",   "command sql.getVersion(clientid:int):lng ",
"mvc_getVersion;",  "Return the database version identifier for a client."  
]
 [ "sql",   "get_value","pattern 

MonetDB: scoping - sql.setVariable is an update instruction, so ...

[Pedro Ferreira]

Changeset: 46d667c9bffd for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=46d667c9bffd
Modified Files:
monetdb5/optimizer/opt_support.c
Branch: scoping
Log Message:

sql.setVariable is an update instruction, so it has side-effects


diffs (13 lines):

diff --git a/monetdb5/optimizer/opt_support.c b/monetdb5/optimizer/opt_support.c
--- a/monetdb5/optimizer/opt_support.c
+++ b/monetdb5/optimizer/opt_support.c
@@ -302,7 +302,8 @@ isUpdateInstruction(InstrPtr p){
getFunctionId(p) == updateRef ||
getFunctionId(p) == deleteRef ||
getFunctionId(p) == growRef ||
-   getFunctionId(p) == clear_tableRef))
+   getFunctionId(p) == clear_tableRef ||
+   getFunctionId(p) == setVariableRef))
return TRUE;
if ( getModuleId(p) == batRef &&
   ( getFunctionId(p) == appendRef ||
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: default - Merged with Jun2020

[Pedro Ferreira]

Changeset: f39eafbd0b4a for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=f39eafbd0b4a
Modified Files:
gdk/gdk.h
gdk/gdk_bat.c
gdk/gdk_bbp.c
gdk/gdk_hash.c
gdk/gdk_imprints.c
gdk/gdk_orderidx.c
gdk/gdk_storage.c
gdk/gdk_system.c
gdk/gdk_system.h
gdk/gdk_utils.c
monetdb5/mal/mal_client.c
monetdb5/mal/mal_dataflow.c
monetdb5/modules/mal/mal_mapi.c
monetdb5/modules/mal/tablet.c
sql/backends/monet5/mal_backend.c
sql/backends/monet5/sql_gencode.c
sql/backends/monet5/sql_statement.c
sql/common/sql_types.c
sql/server/rel_exp.c
sql/server/rel_select.c
sql/server/rel_unnest.c
sql/server/sql_atom.c
sql/server/sql_partition.c
sql/server/sql_query.c
sql/server/sql_semantic.c
sql/storage/bat/bat_storage.c
sql/storage/sql_storage.h
sql/storage/store.c
sql/test/Tests/hot-snapshot.py
sql/test/miscellaneous/Tests/simple_selects.sql
sql/test/miscellaneous/Tests/simple_selects.stable.err
sql/test/subquery/Tests/subquery5.sql
sql/test/subquery/Tests/subquery5.stable.err
sql/test/subquery/Tests/subquery5.stable.out
Branch: default
Log Message:

Merged with Jun2020


diffs (truncated from 879 to 300 lines):

diff --git a/gdk/gdk.h b/gdk/gdk.h
--- a/gdk/gdk.h
+++ b/gdk/gdk.h
@@ -1709,7 +1709,7 @@ typedef struct threadStruct {
 * into this array + 1 (0 is
 * invalid) */
ATOMIC_TYPE pid;/* thread id, 0 = unallocated */
-   char name[16];
+   char name[MT_NAME_LEN];
void *data[THREADDATA];
uintptr_t sp;
 } *Thread;
diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c
--- a/gdk/gdk_bat.c
+++ b/gdk/gdk_bat.c
@@ -121,7 +121,7 @@ BATcreatedesc(oid hseq, int tt, bool hea
bn->tvheap->parentid = bn->batCacheid;
bn->tvheap->farmid = BBPselectfarm(role, bn->ttype, varheap);
}
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "BATlock%d", bn->batCacheid); /* fits */
MT_lock_init(>batIdxLock, name);
bn->batDirtydesc = true;
diff --git a/gdk/gdk_bbp.c b/gdk/gdk_bbp.c
--- a/gdk/gdk_bbp.c
+++ b/gdk/gdk_bbp.c
@@ -910,7 +910,7 @@ BBPreadEntries(FILE *fp, unsigned bbpver
bn->batCount = (BUN) count;
bn->batInserted = bn->batCount;
bn->batCapacity = (BUN) capacity;
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "BATlock%d", bn->batCacheid); /* 
fits */
MT_lock_init(>batIdxLock, name);
 
diff --git a/gdk/gdk_hash.c b/gdk/gdk_hash.c
--- a/gdk/gdk_hash.c
+++ b/gdk/gdk_hash.c
@@ -957,7 +957,7 @@ BAThash(BAT *b)
if (BBP_status(b->batCacheid) & BBPEXISTING && !b->theap.dirty 
&& !GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "hashsync%d", 
b->batCacheid);
MT_lock_unset(>batIdxLock);
if (MT_create_thread(, BAThashsync, b,
diff --git a/gdk/gdk_imprints.c b/gdk/gdk_imprints.c
--- a/gdk/gdk_imprints.c
+++ b/gdk/gdk_imprints.c
@@ -534,7 +534,7 @@ BATimprints(BAT *b)
!GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "impssync%d", 
b->batCacheid);
if (MT_create_thread(, BATimpsync, b,
 MT_THR_DETACHED, name) < 0)
diff --git a/gdk/gdk_orderidx.c b/gdk/gdk_orderidx.c
--- a/gdk/gdk_orderidx.c
+++ b/gdk/gdk_orderidx.c
@@ -169,7 +169,7 @@ persistOIDX(BAT *b)
!GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "oidxsync%d", b->batCacheid);
if (MT_create_thread(, BATidxsync, b,
 MT_THR_DETACHED, name) < 0)
@@ -479,7 +479,7 @@ GDKmergeidx(BAT *b, BAT**a, int n_ar)
b->batInserted == b->batCount) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "oidxsync%d", b->batCacheid);
if (MT_create_thread(, BATidxsync, b,
 MT_THR_DETACHED, name) < 0)
diff --git a/gdk/gdk_storage.c b/gdk/gdk_storage.c
--- a/gdk/gdk_storage.c
+++ b/gdk/gdk_storage.c

MonetDB: scoping - Another simple scoping test file

[Martin Kersten]

Changeset: 8122dad6d26a for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8122dad6d26a
Added Files:
sql/test/scoping/Tests/predefinedvariable.sql
Branch: scoping
Log Message:

Another simple scoping test file


diffs (59 lines):

diff --git a/sql/test/scoping/Tests/predefinedvariable.sql 
b/sql/test/scoping/Tests/predefinedvariable.sql
new file mode 100644
--- /dev/null
+++ b/sql/test/scoping/Tests/predefinedvariable.sql
@@ -0,0 +1,54 @@
+-- MonetDB SQL has a number of predefined global variables, whose semantics 
should be respected.
+
+select optimizer;
+
+-- this one is defined in the sys schema
+select current_schema;
+select sys.optimizer;  -- can we find it there?
+
+select tmp.optimizer;  -- should not be found
+
+-- the optimizer variable can be reassigned a value
+set optimizer = 'minimal_pipe';
+select optimizer;
+
+set sys.optimizer = 'minimal_pipe';
+select sys.optimizer;
+
+-- the global variable may appear as a column in a table
+create table mynewone( i integer, optimizer integer);
+insert into mynewone(i, optimizer) values(1,2);-- to marked as 
sheelding outer definition (ako Pythonic)
+select i, optimizer from mynewone; -- 
ambiguous
+select i, sys.optimizer from mynewone; -- should be 
recognized
+
+-- entering the world of functions
+create function foo()
+returns integer
+begin
+   return optimizer;
+end;
+
+create function foo2()
+returns integer
+begin
+   return sys.optimizer;
+end;
+
+create procedure poo()
+begin
+   set optimizer='volcano_pipe';
+end;
+
+create procedure poo2()
+begin
+   set sys.optimizer='volcano_pipe';   
+end;
+-- returns: syntax error, unexpected END in: "end"
+
+
+create procedure poo4()
+begin
+   set optimizer='deep-pipe';
+   select optimizer;
+end;
+-- returns : syntax error, unexpected END in: "end"
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: mbedded - merged with default

[Niels Nes]

Changeset: 5df7adacf00d for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=5df7adacf00d
Modified Files:
sql/server/rel_select.c
sql/server/rel_unnest.c
sql/server/rel_unnest.h
sql/test/Tests/hot-snapshot.py
sql/test/subquery/Tests/subquery3.stable.err
sql/test/subquery/Tests/subquery5.stable.err
Branch: mbedded
Log Message:

merged with default


diffs (truncated from 329 to 300 lines):

diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c
--- a/sql/server/rel_select.c
+++ b/sql/server/rel_select.c
@@ -1191,7 +1191,7 @@ rel_column_ref(sql_query *query, sql_rel
if (exp)
break;
}
-   if (exp && outer && outer->card <= CARD_AGGR && 
exp->card > CARD_AGGR && (!is_sql_aggr(f) || is_sql_farg(f)))
+   if (exp && outer && outer->card <= CARD_AGGR && 
exp->card > CARD_AGGR && !is_sql_aggr(f))
return sql_error(sql, ERR_GROUPBY, 
SQLSTATE(42000) "SELECT: cannot use non GROUP BY column '%s' in query results 
without an aggregate function", name);
if (exp && outer && !is_sql_aggr(f)) {
if (query_outer_used_exp( query, i, exp, f)) {
@@ -1257,7 +1257,7 @@ rel_column_ref(sql_query *query, sql_rel
if (exp)
break;
}
-   if (exp && outer && outer->card <= CARD_AGGR && 
exp->card > CARD_AGGR && (!is_sql_aggr(f) || is_sql_farg(f)))
+   if (exp && outer && outer->card <= CARD_AGGR && 
exp->card > CARD_AGGR && !is_sql_aggr(f))
return sql_error(sql, ERR_GROUPBY, 
SQLSTATE(42000) "SELECT: cannot use non GROUP BY column '%s.%s' in query 
results without an aggregate function", tname, cname);
if (exp && outer && !is_sql_aggr(f)) {
if (query_outer_used_exp( query, i, exp, f)) {
@@ -3349,14 +3349,16 @@ static sql_exp *
 
exps = sa_list(sql->sa);
if (args && args->data.sym) {
-   int ungrouped_col = -1, i, all_aggr = query_has_outer(query);
+   int i, all_aggr = query_has_outer(query);
+   bool found_nested_aggr = false;
+   list *ungrouped_cols = NULL;
+
all_freevar = 1;
-   bool found_nested_aggr = false;
for (i = 0; args && args->data.sym; args = args->next, i++) {
int base = (!groupby || !is_project(groupby->op) || 
is_base(groupby->op) || is_processed(groupby));
-   bool found_one = false;
-   sql_rel *outer = NULL, *gl = base?groupby:groupby->l, 
*ogl = gl; /* handle case of subqueries without correlation */
-   sql_exp *e = rel_value_exp(query, , args->data.sym, 
(f | sql_aggr)& ~sql_farg, ek), *a = NULL;
+   sql_rel *gl = base?groupby:groupby->l, *ogl = gl; /* 
handle case of subqueries without correlation */
+   sql_exp *e = rel_value_exp(query, , args->data.sym, 
(f | sql_aggr)& ~sql_farg, ek);
+   bool found_one_freevar = false;
 
has_args = true;
if (gl && gl != ogl) {
@@ -3384,38 +3386,15 @@ static sql_exp *
return e;
}
 
-   if (is_freevar(e) && e->type == e_column) {
-   if ((outer = query_fetch_outer(query, 
is_freevar(e)-1))) {
-   if ((a = rel_find_exp(outer, e)) && 
is_aggr(a->type))
-   return sql_error(sql, 05, 
SQLSTATE(42000) "SELECT: aggregate function calls cannot be nested");
-   }
-   }
-
-   if (all_aggr) {
-   /* get expression from outer */
-   int aggr = 0;
-   if (a)
-   aggr = is_aggr(a->type);
-   else if (outer && outer->grouped)
-   ungrouped_col = i;
-   all_aggr &= aggr;
-   } else {
-   all_aggr &= (exp_card(e) <= CARD_AGGR && 
!exp_is_atom(e) && is_aggr(e->type) && !is_func(e->type) && (!groupby || 
!is_groupby(groupby->op) || !groupby->r || !exps_find_exp(groupby->r, e)));
-   }
-   all_freevar &= (exp_only_freevar(query, e, _one, 
_nested_aggr) && found_one);
+   all_aggr &= (exp_card(e) <= CARD_AGGR && 
!exp_is_atom(e) && is_aggr(e->type) && !is_func(e->type) && (!groupby || 
!is_groupby(groupby->op) || !groupby->r || 

MonetDB: unlock - keep number of deleted catalog entries

[Niels Nes]

Changeset: 9ca9c41a54a9 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9ca9c41a54a9
Modified Files:
gdk/gdk_logger.c
gdk/gdk_logger_internals.h
Branch: unlock
Log Message:

keep number of deleted catalog entries


diffs (52 lines):

diff --git a/gdk/gdk_logger.c b/gdk/gdk_logger.c
--- a/gdk/gdk_logger.c
+++ b/gdk/gdk_logger.c
@@ -1173,7 +1173,7 @@ logger_switch_bat(BAT *old, BAT *new, co
 static gdk_return
 bm_get_counts(logger *lg)
 {
-   BUN p, q;
+   BUN p, q, deleted = 0;
const log_bid *bids = (const log_bid *) Tloc(lg->catalog_bid, 0);
 
BATloop(lg->catalog_bid, p, q) {
@@ -1183,10 +1183,14 @@ bm_get_counts(logger *lg)
if (BUNfnd(lg->dcatalog, ) == BUN_NONE) {
BAT *b = BBPquickdesc(bids[p], 1);
cnt = BATcount(b);
+   } else {
+   deleted++;
}
if (BUNappend(lg->catalog_cnt, , false) != GDK_SUCCEED)
return GDK_FAIL;
}
+   lg->deleted = deleted;
+   lg->cnt = BATcount(lg->catalog_bid);
return GDK_SUCCEED;
 }
 
@@ -1226,9 +1230,9 @@ bm_subcommit(logger *lg)
n[i++] = col;
}
/* now commit catalog, so it's also up to date on disk */
-   sizes[i] = (BUN)lg->cnt;
+   sizes[i] = lg->cnt;
n[i++] = catalog_bid->batCacheid;
-   sizes[i] = (BUN)lg->cnt;
+   sizes[i] = lg->cnt;
n[i++] = catalog_id->batCacheid;
sizes[i] = BATcount(dcatalog); /* todo ! */
n[i++] = dcatalog->batCacheid;
diff --git a/gdk/gdk_logger_internals.h b/gdk/gdk_logger_internals.h
--- a/gdk/gdk_logger_internals.h
+++ b/gdk/gdk_logger_internals.h
@@ -36,7 +36,8 @@ struct logger {
BAT *catalog_id;/* object identifier is unique */
BAT *catalog_cnt;   /* count of ondisk buns (transient) */
BAT *dcatalog;  /* deleted from catalog table */
-   lng cnt;/* number of persistent bats, incremented on 
log flushing */
+   BUN cnt;/* number of persistent bats, incremented on 
log flushing */
+   BUN deleted;/* number of destroyed persistent bats, needed 
for catalog vacuum */
 
BAT *seqs_id;   /* int id column */
BAT *seqs_val;  /* lng value column */
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: scoping - Added missing bindings to temp tables if it w...

[Pedro Ferreira]

Changeset: 3ad89a78c61a for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3ad89a78c61a
Modified Files:
sql/server/rel_psm.c
sql/server/rel_schema.c
sql/server/sql_privileges.c
Branch: scoping
Log Message:

Added missing bindings to temp tables if it was the case


diffs (199 lines):

diff --git a/sql/server/rel_psm.c b/sql/server/rel_psm.c
--- a/sql/server/rel_psm.c
+++ b/sql/server/rel_psm.c
@@ -474,19 +474,26 @@ rel_psm_return( sql_query *query, sql_su
 
if (!sname && (t = stack_find_table(sql, tname))) {
rel = rel_table(sql, ddl_create_table, s->base.name, t, 
SQL_DECLARED_TABLE);
-   } else if ((t = mvc_bind_table(sql, s, tname))) {
-   rel = rel_basetable(sql, t, t->base.name);
-   for (node *n = rel->exps->h ; n ; n = n->next) {
-   sql_exp *e = (sql_exp *) n->data;   
-   const char *oname = e->r;
+   } else {
+   t = mvc_bind_table(sql, s, tname);
+   if (!t && !sname) {
+   s = tmp_schema(sql);
+   t = mvc_bind_table(sql, s, tname);
+   }
+   if (t) {
+   rel = rel_basetable(sql, t, t->base.name);
+   for (node *n = rel->exps->h ; n ; n = n->next) {
+   sql_exp *e = (sql_exp *) n->data;
+   const char *oname = e->r;
 
-   if (!strcmp(oname, TID)) {
-   list_remove_node(rel->exps, n);
-   break;
+   if (!strcmp(oname, TID)) {
+   list_remove_node(rel->exps, n);
+   break;
+   }
}
-   }
-   } else
-   return sql_error(sql, 02, SQLSTATE(42S02) "RETURN: no 
such table '%s'", tname);
+   } else
+   return sql_error(sql, 02, SQLSTATE(42S02) 
"RETURN: no such table '%s'", tname);
+   }
} else { /* other cases */
res = rel_value_exp2(query, , return_sym, sql_sel, ek);
if (!res)
@@ -1309,7 +1316,12 @@ create_trigger(sql_query *query, dlist *
t = stack_find_table(sql, tname);
if (t)
return sql_error(sql, 02, SQLSTATE(42000) "%s TRIGGER: 
declared tables cannot have triggers", base);
-   if (!(t = mvc_bind_table(sql, ss, tname)))
+   t = mvc_bind_table(sql, ss, tname);
+   if (!t && !sname) {
+   ss = tmp_schema(sql);
+   t = mvc_bind_table(sql, ss, tname);
+   }
+   if (!t)
return sql_error(sql, 02, SQLSTATE(42000) "%s TRIGGER: 
unknown table '%s'", base, tname);
}
if (create && isView(t))
@@ -1539,6 +1551,10 @@ create_table_from_loader(sql_query *quer
t = stack_find_table(sql, tname);
if (!t)
t = mvc_bind_table(sql, s, tname);
+   if (!t && !sname) {
+   s = tmp_schema(sql);
+   t = mvc_bind_table(sql, s, tname);
+   }
if (t)
return sql_error(sql, 02, SQLSTATE(42S01) "CREATE TABLE FROM 
LOADER: name '%s' already in use", tname);
 
diff --git a/sql/server/rel_schema.c b/sql/server/rel_schema.c
--- a/sql/server/rel_schema.c
+++ b/sql/server/rel_schema.c
@@ -526,6 +526,13 @@ table_foreign_key(mvc *sql, char *name, 
return SQL_ERR;
}
ft = mvc_bind_table(sql, fs, rtname);
+   if (!ft && !rsname) {
+   sql_schema *save = fs;
+   fs = tmp_schema(sql);
+   ft = mvc_bind_table(sql, fs, name);
+   if (!ft)
+   fs = save;
+   }
/* self referenced table */
if (!ft && t->s == fs && strcmp(t->base.name, rtname) == 0)
ft = t;
@@ -866,6 +873,10 @@ table_element(sql_query *query, symbol *
ot = stack_find_table(sql, name);
if (!ot)
ot = mvc_bind_table(sql, os, name);
+   if (!ot && !sname) {
+   os = tmp_schema(sql);
+   ot = mvc_bind_table(sql, os, name);
+   }
if (!ot) {
sql_error(sql, 02, SQLSTATE(3F000) "%s: no such table 
'%s'", action, name);
return SQL_ERR;
@@ -1156,6 +1167,13 @@ rel_create_view(sql_query *query, sql_sc
t = stack_find_table(sql, name);
if 

MonetDB: Jun2020 - Merge with Nov2019 branch.

[Sjoerd Mullender]

Changeset: 1dc2d13303f7 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1dc2d13303f7
Modified Files:
sql/server/sql_partition.c
sql/server/sql_semantic.c
sql/storage/bat/bat_storage.c
sql/storage/sql_storage.h
sql/storage/store.c
Branch: Jun2020
Log Message:

Merge with Nov2019 branch.

___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: scoping - Fixed error message and remove names for sele...

[Pedro Ferreira]

Changeset: 92b70c48d7e1 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=92b70c48d7e1
Modified Files:
sql/server/rel_schema.c
sql/server/rel_select.c
sql/test/pg_regress/Tests/alter_table.stable.err
Branch: scoping
Log Message:

Fixed error message and remove names for select and with frames. They were 
redundant


diffs (85 lines):

diff --git a/sql/server/rel_schema.c b/sql/server/rel_schema.c
--- a/sql/server/rel_schema.c
+++ b/sql/server/rel_schema.c
@@ -1416,7 +1416,7 @@ get_schema_name( mvc *sql, char *sname, 
if (!sname) {
sql_schema *ss = cur_schema(sql);
sql_table *t = mvc_bind_table(sql, ss, tname);
-   if (!t)
+   if (!t && !sname)
ss = tmp_schema(sql);
sname = ss->base.name;
}
@@ -1443,6 +1443,13 @@ sql_alter_table(sql_query *query, dlist 
t = stack_find_table(sql, tname);
if (!t)
t = mvc_bind_table(sql, s, tname);
+   if (!t && !sname) {
+   sql_schema *save = s;
+   s = tmp_schema(sql);
+   t = mvc_bind_table(sql, s, tname);
+   if (!t)
+   s = save;
+   }
if (!t) {
if (if_exists)
return rel_psm_block(sql->sa, new_exp_list(sql->sa));
@@ -1450,9 +1457,9 @@ sql_alter_table(sql_query *query, dlist 
}
 
if (isDeclaredTable(t))
-   return sql_error(sql, 02, SQLSTATE(42000) "ALTER TABLE: can't 
alter declared table '%s'", tname);
+   return sql_error(sql, 02, SQLSTATE(42S02) "ALTER TABLE: can't 
alter declared table '%s'", tname);
if (isTempSchema(t->s))
-   return sql_error(sql, 02, SQLSTATE(42000) "ALTER TABLE: can't 
alter temporary table '%s'", tname);
+   return sql_error(sql, 02, SQLSTATE(42S02) "ALTER TABLE: can't 
alter temporary table '%s'", tname);
 
assert(te);
if (t->persistence != SQL_DECLARED_TABLE)
diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c
--- a/sql/server/rel_select.c
+++ b/sql/server/rel_select.c
@@ -251,7 +251,7 @@ rel_with_query(sql_query *query, symbol 
symbol *next = d->next->data.sym;
sql_rel *rel;
 
-   if (!stack_push_frame(sql, "WITH"))
+   if (!stack_push_frame(sql, NULL))
return sql_error(sql, 02, SQLSTATE(HY013) MAL_MALLOC_FAIL);
/* first handle all with's (ie inlined views) */
for (d = d->data.lval->h; d; d = d->next) {
@@ -6013,7 +6013,7 @@ rel_subquery(sql_query *query, sql_rel *
mvc *sql = query->sql;
int toplevel = 0;
 
-   if (!stack_push_frame(sql, "SELECT"))
+   if (!stack_push_frame(sql, NULL))
return sql_error(sql, 02, SQLSTATE(HY013) MAL_MALLOC_FAIL);
 
if (!rel || (rel->op == op_project &&
diff --git a/sql/test/pg_regress/Tests/alter_table.stable.err 
b/sql/test/pg_regress/Tests/alter_table.stable.err
--- a/sql/test/pg_regress/Tests/alter_table.stable.err
+++ b/sql/test/pg_regress/Tests/alter_table.stable.err
@@ -76,11 +76,11 @@ ERROR = !Create Key failed, key 'tmpcons
 CODE  = 42000
 MAPI  = (monetdb) /var/tmp/mtest-30274/.s.monetdb.37685
 QUERY = ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references pktable;
-ERROR = !ALTER TABLE: not supported on TEMPORARY table 'fktable'
+ERROR = !ALTER TABLE: can't alter temporary table 'fktable'
 CODE  = 42S02
 MAPI  = (monetdb) /var/tmp/mtest-30274/.s.monetdb.37685
 QUERY = ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references pktable(ptest1);
-ERROR = !ALTER TABLE: not supported on TEMPORARY table 'fktable'
+ERROR = !ALTER TABLE: can't alter temporary table 'fktable'
 CODE  = 42S02
 MAPI  = (monetdb) /var/tmp/mtest-30274/.s.monetdb.37685
 QUERY = ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references pktable(ptest1);
@@ -770,7 +770,7 @@ ERROR = !syntax error, unexpected IDENT 
 CODE  = 42000
 MAPI  = (monetdb) /var/tmp/mtest-30274/.s.monetdb.37685
 QUERY = alter table foo drop column f2;
-ERROR = !ALTER TABLE: not supported on TEMPORARY table 'foo'
+ERROR = !ALTER TABLE: can't alter temporary table 'foo'
 CODE  = 42S02
 MAPI  = (monetdb) /var/tmp/mtest-30274/.s.monetdb.37685
 QUERY = alter table foo alter f1 TYPE integer; -- fails
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Jun2020 - Don't print server error to stdout, use gdk t...

[Pedro Ferreira]

Changeset: ba39a301566e for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ba39a301566e
Modified Files:
sql/server/rel_select.c
Branch: Jun2020
Log Message:

Don't print server error to stdout, use gdk tracer instead


diffs (12 lines):

diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c
--- a/sql/server/rel_select.c
+++ b/sql/server/rel_select.c
@@ -1375,7 +1375,7 @@ exp_fix_scale(mvc *sql, sql_subtype *ct,
res->scale = 0;
return exp_binop(sql->sa, e, exp_atom(sql->sa, a), c);
} else {
-   printf("scale_down missing (%s)\n", 
et->type->base.name);
+   TRC_CRITICAL(SQL_PARSER, "scale_down missing (%s)\n", 
et->type->base.name);
}
}
return e;
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Fix more whitespace

[Joeri van Ruth]

Changeset: 2a018ce3c282 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=2a018ce3c282
Modified Files:
clients/examples/C/streamcat.c
common/stream/Tests/testdata.py
common/stream/Tests/write_tests.py
Branch: makelibstreamgreatagain
Log Message:

Fix more whitespace


diffs (33 lines):

diff --git a/clients/examples/C/streamcat.c b/clients/examples/C/streamcat.c
--- a/clients/examples/C/streamcat.c
+++ b/clients/examples/C/streamcat.c
@@ -237,4 +237,4 @@ opener_wastream(char *filename)
if (s == NULL)
croak(2, "Error opening file '%s': %s", filename, 
strerror(errno));
return s;
-}
\ No newline at end of file
+}
diff --git a/common/stream/Tests/testdata.py b/common/stream/Tests/testdata.py
--- a/common/stream/Tests/testdata.py
+++ b/common/stream/Tests/testdata.py
@@ -159,4 +159,4 @@ class Doc:
 if __name__ == "__main__":
 d = Doc("banana")
 p = d.write_tmp()
-print(f"path to doc {d.name} is {p}")
\ No newline at end of file
+print(f"path to doc {d.name} is {p}")
diff --git a/common/stream/Tests/write_tests.py 
b/common/stream/Tests/write_tests.py
--- a/common/stream/Tests/write_tests.py
+++ b/common/stream/Tests/write_tests.py
@@ -101,6 +101,6 @@ def all_tests(filename_filter):
 
 
 # if __name__ == "__main__":
-#  docname = "small.txt.gz"
-#  doc = [d for d in gen_docs()][0]
-#  test_write('rastream', True, doc)
+#   docname = "small.txt.gz"
+#   doc = [d for d in gen_docs()][0]
+#   test_write('rastream', True, doc)
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Fix whitespace

[Joeri van Ruth]

Changeset: a8cf14bec2b9 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a8cf14bec2b9
Modified Files:
common/stream/Tests/read_bz2.py
common/stream/Tests/read_gz.py
common/stream/Tests/read_lz4.py
common/stream/Tests/read_tests.py
common/stream/Tests/read_uncompressed.py
common/stream/Tests/read_xz.py
common/stream/Tests/write_bz2.py
common/stream/Tests/write_gz.py
common/stream/Tests/write_lz4.py
common/stream/Tests/write_uncompressed.py
common/stream/Tests/write_xz.py
Branch: makelibstreamgreatagain
Log Message:

Fix whitespace


diffs (truncated from 305 to 300 lines):

diff --git a/common/stream/Tests/read_bz2.py b/common/stream/Tests/read_bz2.py
--- a/common/stream/Tests/read_bz2.py
+++ b/common/stream/Tests/read_bz2.py
@@ -3,7 +3,10 @@
 import read_tests
 import sys
 
-filter = lambda f: f.endswith('.txt.bz2')
+
+def filter(f):
+return f.endswith('.txt.bz2')
+
 
 if read_tests.all_tests(filter) != 0:
-   sys.exit(1)
+sys.exit(1)
diff --git a/common/stream/Tests/read_gz.py b/common/stream/Tests/read_gz.py
--- a/common/stream/Tests/read_gz.py
+++ b/common/stream/Tests/read_gz.py
@@ -3,7 +3,10 @@
 import read_tests
 import sys
 
-filter = lambda f: f.endswith('.txt.gz')
+
+def filter(f):
+return f.endswith('.txt.gz')
+
 
 if read_tests.all_tests(filter) != 0:
-   sys.exit(1)
+sys.exit(1)
diff --git a/common/stream/Tests/read_lz4.py b/common/stream/Tests/read_lz4.py
--- a/common/stream/Tests/read_lz4.py
+++ b/common/stream/Tests/read_lz4.py
@@ -3,7 +3,10 @@
 import read_tests
 import sys
 
-filter = lambda f: f.endswith('.txt.lz4')
+
+def filter(f):
+return f.endswith('.txt.lz4')
+
 
 if read_tests.all_tests(filter) != 0:
-   sys.exit(1)
+sys.exit(1)
diff --git a/common/stream/Tests/read_tests.py 
b/common/stream/Tests/read_tests.py
--- a/common/stream/Tests/read_tests.py
+++ b/common/stream/Tests/read_tests.py
@@ -13,85 +13,87 @@ BOM = b'\xEF\xBB\xBF'
 
 
 def gen_compr_variants(name, content, limit):
-   yield testdata.Doc(name, content, limit, None)
-   yield testdata.Doc(name + ".gz", content, limit, "gz")
-   yield testdata.Doc(name + ".bz2", content, limit, "bz2")
-   yield testdata.Doc(name + ".xz", content, limit, "xz")
-   yield testdata.Doc(name + ".lz4", content, limit, "lz4")
+yield testdata.Doc(name, content, limit, None)
+yield testdata.Doc(name + ".gz", content, limit, "gz")
+yield testdata.Doc(name + ".bz2", content, limit, "bz2")
+yield testdata.Doc(name + ".xz", content, limit, "xz")
+yield testdata.Doc(name + ".lz4", content, limit, "lz4")
 
 
 def gen_bom_compr_variants(name, content, limit):
-   yield from gen_compr_variants(name + ".txt", content, limit)
-   yield from gen_compr_variants(name + "_bom.txt", BOM + content, limit)
+yield from gen_compr_variants(name + ".txt", content, limit)
+yield from gen_compr_variants(name + "_bom.txt", BOM + content, limit)
 
 
 def gen_docs():
-   input = testdata.SHERLOCK
+input = testdata.SHERLOCK
 
-   # Whole file
-   yield from gen_bom_compr_variants('sherlock', input, None)
+# Whole file
+yield from gen_bom_compr_variants('sherlock', input, None)
 
-   # Empty file
-   yield from gen_bom_compr_variants('empty', b'', None)
+# Empty file
+yield from gen_bom_compr_variants('empty', b'', None)
 
-   # First 16 lines
-   head = b'\n'.join(input.split(b'\n')[:16]) + b'\n'
-   yield from gen_bom_compr_variants('small', head, None)
+# First 16 lines
+head = b'\n'.join(input.split(b'\n')[:16]) + b'\n'
+yield from gen_bom_compr_variants('small', head, None)
 
-   # Buffer size boundary cases
-   for base_size in [1024, 2048, 4096, 8192, 16384]:
-   for delta in [-1, 0, 1]:
-   size = base_size + delta
-   yield from gen_bom_compr_variants(f'block{size}', 
input, size)
+# Buffer size boundary cases
+for base_size in [1024, 2048, 4096, 8192, 16384]:
+for delta in [-1, 0, 1]:
+size = base_size + delta
+yield from gen_bom_compr_variants(f'block{size}', input, size)
 
 
 def test_read(opener, text_mode, doc):
-   filename = doc.write_tmp()
-   test = f"read {opener} {doc.name}"
-
-   print()
+filename = doc.write_tmp()
+test = f"read {opener} {doc.name}"
 
-   cmd = [ 'streamcat', 'read', filename, opener ]
-   results = subprocess.run(cmd, stdout=subprocess.PIPE, 
stderr=subprocess.PIPE)
-   if results.returncode != 0 or results.stderr:
-   print(f"Test {test}: streamcat returned with exit code 
{results.returncode}:\n{results.stderr or ''}")
-   return False
+print()
 
-   output = results.stdout or b""
-   complaint = doc.verify(output, text_mode)
+cmd = ['streamcat', 'read', filename, opener]
+results = 

MonetDB: makelibstreamgreatagain - Start splitting stream.c into...

[Joeri van Ruth]

Changeset: d6aca87ba535 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=d6aca87ba535
Added Files:
common/stream/bz2_stream.c
common/stream/gz_stream.c
common/stream/iconv_stream.c
common/stream/lz4_stream.c
common/stream/misc.c
common/stream/oldcode.c
common/stream/socket_stream.c
common/stream/stdio_stream.c
common/stream/stream_internal.h
common/stream/url_stream.c
common/stream/xz_stream.c
Modified Files:
common/stream/Makefile.ag
common/stream/stream.c
Branch: makelibstreamgreatagain
Log Message:

Start splitting stream.c into separate parts


diffs (truncated from 11073 to 300 lines):

diff --git a/common/stream/Makefile.ag b/common/stream/Makefile.ag
--- a/common/stream/Makefile.ag
+++ b/common/stream/Makefile.ag
@@ -17,7 +17,13 @@ INCLUDES = ../utils \
   $(curl_CFLAGS)
 
 lib_stream  =  {
-   SOURCES = stream.c stream.h stream_socket.h
+   SOURCES = oldcode.c stream.c \
+   stdio_stream.c \
+   gz_stream.c bz2_stream.c xz_stream.c lz4_stream.c \
+   url_stream.c socket_stream.c \
+   iconv_stream.c \
+   misc.c \
+   stream.h stream_internal.h stream_socket.h
VERSION = $(STREAM_VERSION)
LIBS = WIN32?$(SOCKET_LIBS) \
   WIN32?$(zlib_LIBS) \
diff --git a/common/stream/bz2_stream.c b/common/stream/bz2_stream.c
new file mode 100644
--- /dev/null
+++ b/common/stream/bz2_stream.c
@@ -0,0 +1,273 @@
+/*
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0.  If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ *
+ * Copyright 1997 - July 2008 CWI, August 2008 - 2020 MonetDB B.V.
+ */
+
+/* streams working on a bzip2-compressed disk file */
+
+#include "monetdb_config.h"
+#include "stream.h"
+#include "stream_internal.h"
+
+
+#ifdef HAVE_LIBBZ2
+struct bz {
+   BZFILE *b;
+   FILE *f;
+};
+
+static void
+stream_bzclose(stream *s)
+{
+   int err = BZ_OK;
+
+   if (s->stream_data.p) {
+   if (s->readonly)
+   BZ2_bzReadClose(, ((struct bz *) 
s->stream_data.p)->b);
+   else
+   BZ2_bzWriteClose(, ((struct bz *) 
s->stream_data.p)->b, 0, NULL, NULL);
+   fclose(((struct bz *) s->stream_data.p)->f);
+   free(s->stream_data.p);
+   }
+   s->stream_data.p = NULL;
+}
+
+static ssize_t
+stream_bzread(stream *restrict s, void *restrict buf, size_t elmsize, size_t 
cnt)
+{
+   size_t size = elmsize * cnt;
+   int err;
+   void *punused;
+   int nunused;
+   char unused[BZ_MAX_UNUSED];
+   struct bz *bzp = s->stream_data.p;
+
+   if (bzp == NULL) {
+   s->errnr = MNSTR_READ_ERROR;
+   return -1;
+   }
+   if (size == 0)
+   return 0;
+   size = (size_t) BZ2_bzRead(, bzp->b, buf, size > ((size_t) 1 << 30) 
? 1 << 30 : (int) size);
+   if (err == BZ_STREAM_END) {
+   /* end of stream, but not necessarily end of file: get
+* unused bits, close stream, and open again with the
+* saved unused bits */
+   BZ2_bzReadGetUnused(, bzp->b, , );
+   if (err == BZ_OK && (nunused > 0 || !feof(bzp->f))) {
+   if (nunused > 0)
+   memcpy(unused, punused, nunused);
+   BZ2_bzReadClose(, bzp->b);
+   bzp->b = BZ2_bzReadOpen(, bzp->f, 0, 0, unused, 
nunused);
+   } else {
+   stream_bzclose(s);
+   }
+   }
+   if (err != BZ_OK) {
+   s->errnr = MNSTR_READ_ERROR;
+   return -1;
+   }
+   /* when in text mode, convert \r\n line endings to \n */
+   if (!s->binary) {
+   char *p1, *p2, *pe;
+
+   p1 = buf;
+   pe = p1 + size;
+   while (p1 < pe && *p1 != '\r')
+   p1++;
+   p2 = p1;
+   while (p1 < pe) {
+   if (*p1 == '\r' && p1[1] == '\n')
+   size--;
+   else
+   *p2++ = *p1;
+   p1++;
+   }
+   }
+   return (ssize_t) (size / elmsize);
+}
+
+static ssize_t
+stream_bzwrite(stream *restrict s, const void *restrict buf, size_t elmsize, 
size_t cnt)
+{
+   size_t size = elmsize * cnt;
+   int err;
+   struct bz *bzp = s->stream_data.p;
+
+   if (bzp == NULL) {
+   s->errnr = MNSTR_WRITE_ERROR;
+   return -1;
+   }
+   if (size == 0)
+   return 0;
+   while (size > 0) {
+   int sz = size > (1 << 30) ? 1 << 30 : (int) size;
+   BZ2_bzWrite(, bzp->b, (void *) 

MonetDB: makelibstreamgreatagain - Add write tests

[Joeri van Ruth]

Changeset: 94104b893fc5 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=94104b893fc5
Added Files:
common/stream/Tests/read_tests.py
common/stream/Tests/write_bz2.py
common/stream/Tests/write_bz2.stable.err
common/stream/Tests/write_bz2.stable.out
common/stream/Tests/write_gz.py
common/stream/Tests/write_gz.stable.err
common/stream/Tests/write_gz.stable.out
common/stream/Tests/write_lz4.py
common/stream/Tests/write_lz4.stable.err
common/stream/Tests/write_lz4.stable.out
common/stream/Tests/write_tests.py
common/stream/Tests/write_uncompressed.py
common/stream/Tests/write_uncompressed.stable.err
common/stream/Tests/write_uncompressed.stable.out
common/stream/Tests/write_xz.py
common/stream/Tests/write_xz.stable.err
common/stream/Tests/write_xz.stable.out
Removed Files:
common/stream/Tests/streamtests.py
Modified Files:
common/stream/Tests/All
common/stream/Tests/read_bz2.py
common/stream/Tests/read_gz.py
common/stream/Tests/read_lz4.py
common/stream/Tests/read_uncompressed.py
common/stream/Tests/read_xz.py
common/stream/Tests/testdata.py
Branch: makelibstreamgreatagain
Log Message:

Add write tests


diffs (truncated from 882 to 300 lines):

diff --git a/common/stream/Tests/All b/common/stream/Tests/All
--- a/common/stream/Tests/All
+++ b/common/stream/Tests/All
@@ -1,5 +1,11 @@
 read_uncompressed
-read_bz2
-read_gz
-read_lz4
-read_xz
+HAVE_LIBBZ2?read_bz2
+HAVE_LIBZ?read_gz
+HAVE_LIBLZMA?read_lz4
+HAVE_LIBLZ4?read_xz
+
+write_uncompressed
+HAVE_LIBBZ2?write_bz2
+HAVE_LIBZ?write_gz
+HAVE_LIBLZMA?write_lz4
+HAVE_LIBLZ4?write_xz
diff --git a/common/stream/Tests/read_bz2.py b/common/stream/Tests/read_bz2.py
--- a/common/stream/Tests/read_bz2.py
+++ b/common/stream/Tests/read_bz2.py
@@ -1,9 +1,9 @@
 #!/usr/bin/env python3
 
-import streamtests
+import read_tests
 import sys
 
 filter = lambda f: f.endswith('.txt.bz2')
 
-if streamtests.all_read_tests(filter) != 0:
+if read_tests.all_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/read_gz.py b/common/stream/Tests/read_gz.py
--- a/common/stream/Tests/read_gz.py
+++ b/common/stream/Tests/read_gz.py
@@ -1,9 +1,9 @@
 #!/usr/bin/env python3
 
-import streamtests
+import read_tests
 import sys
 
 filter = lambda f: f.endswith('.txt.gz')
 
-if streamtests.all_read_tests(filter) != 0:
+if read_tests.all_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/read_lz4.py b/common/stream/Tests/read_lz4.py
--- a/common/stream/Tests/read_lz4.py
+++ b/common/stream/Tests/read_lz4.py
@@ -1,9 +1,9 @@
 #!/usr/bin/env python3
 
-import streamtests
+import read_tests
 import sys
 
 filter = lambda f: f.endswith('.txt.lz4')
 
-if streamtests.all_read_tests(filter) != 0:
+if read_tests.all_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/streamtests.py 
b/common/stream/Tests/read_tests.py
old mode 100755
new mode 100644
rename from common/stream/Tests/streamtests.py
rename to common/stream/Tests/read_tests.py
--- a/common/stream/Tests/streamtests.py
+++ b/common/stream/Tests/read_tests.py
@@ -81,29 +81,16 @@ def test_reads(doc):
return failures
 
 
-def all_read_tests(filename_filter):
+def all_tests(filename_filter):
failures = 0
for d in gen_docs():
if not filename_filter(d.name):
continue
-
-   # rstream does not strip BOM
failures += test_reads(d)
 
return failures
 
 
-def main(kind, file):
-   failures = 0
-   if kind == "read":
-   docs = [d for d in gen_docs() if d.filename == file]
-   print(f"Found {len(docs)} matches.")
-   print()
-   for d in docs:
-   failures += test_reads(d)
-
-   return failures == 0
-
 if __name__ == "__main__":
docname = "small.txt.gz"
doc = [d for d in gen_docs()][0]
diff --git a/common/stream/Tests/read_uncompressed.py 
b/common/stream/Tests/read_uncompressed.py
--- a/common/stream/Tests/read_uncompressed.py
+++ b/common/stream/Tests/read_uncompressed.py
@@ -1,9 +1,9 @@
 #!/usr/bin/env python3
 
-import streamtests
+import read_tests
 import sys
 
 filter = lambda f: f.endswith('.txt')
 
-if streamtests.all_read_tests(filter) != 0:
+if read_tests.all_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/read_xz.py b/common/stream/Tests/read_xz.py
--- a/common/stream/Tests/read_xz.py
+++ b/common/stream/Tests/read_xz.py
@@ -1,9 +1,9 @@
 #!/usr/bin/env python3
 
-import streamtests
+import read_tests
 import sys
 
 filter = lambda f: f.endswith('.txt.xz')
 
-if streamtests.all_read_tests(filter) != 0:
+if read_tests.all_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/testdata.py b/common/stream/Tests/testdata.py
--- 

MonetDB: makelibstreamgreatagain - Add write support to streamcat

[Joeri van Ruth]

Changeset: 1c1c41af84af for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1c1c41af84af
Modified Files:
clients/examples/C/streamcat.c
common/stream/Tests/testdata.py
Branch: makelibstreamgreatagain
Log Message:

Add write support to streamcat


diffs (179 lines):

diff --git a/clients/examples/C/streamcat.c b/clients/examples/C/streamcat.c
--- a/clients/examples/C/streamcat.c
+++ b/clients/examples/C/streamcat.c
@@ -10,19 +10,30 @@
 
 const char *USAGE =
"Usage:\n"
-   "streamcat read FILENAME OPENER\n"
-   "With OPENER:\n"
+   "streamcat read  FILENAME R_OPENER\n"
+   "streamcat write FILENAME W_OPENER\n"
+   "With R_OPENER:\n"
"- rstream   stream = open_rstream(filename)\n"
"- rastream  stream = open_rastream(filename)\n"
+   "With W_OPENER:\n"
+   "- wstream   stream = open_wstream(filename)\n"
+   "- wastream  stream = open_wastream(filename)\n"
;
 
 
 static int cmd_read(char *argv[]);
+static int cmd_write(char *argv[]);
 typedef stream *(*opener_fun)(char *filename);
+
 static stream *opener_rstream(char *filename);
 static stream *opener_rastream(char *filename);
 
-static void copyout(stream *in);
+static stream *opener_wstream(char *filename);
+static stream *opener_wastream(char *filename);
+
+static void copy_to_stdout(stream *in);
+
+static void copy_from_stdin(stream *out);
 
 _Noreturn static void croak(int status, const char *msg, ...)
__attribute__((__format__(__printf__, 2, 3)));
@@ -56,6 +67,8 @@ main(int argc, char *argv[])
 
if (strcmp(argv[1], "read") == 0)
return cmd_read(argv+1);
+   else if (strcmp(argv[1], "write") == 0)
+   return cmd_write(argv+1);
else
croak(1, "Unknown subcommand '%s'", argv[1]);
 }
@@ -86,13 +99,46 @@ int cmd_read(char *argv[])
 
s = opener(filename);
 
-   copyout(s);
+   copy_to_stdout(s);
+   mnstr_close(s);
 
return 0;
 }
 
 
-static void copyout(stream *in)
+int cmd_write(char *argv[])
+{
+   char **arg = [1];
+   char *filename = NULL;
+   char *opener_name = NULL;
+   opener_fun opener;
+
+   stream *s = NULL;
+
+   filename = *arg++;
+   if (filename == NULL)
+   croak(1, "Missing filename");
+
+   opener_name = *arg++;
+   if (opener_name == NULL)
+   croak(1, "Missing opener");
+   else if (strcmp(opener_name, "wstream") == 0)
+   opener = opener_wstream;
+   else if (strcmp(opener_name, "wastream") == 0)
+   opener = opener_wastream;
+   else
+   croak(1, "Unknown opener '%s'", opener_name);
+
+   s = opener(filename);
+
+   copy_from_stdin(s);
+   mnstr_close(s);
+
+   return 0;
+}
+
+
+static void copy_to_stdout(stream *in)
 {
FILE *out;
char buf[1024];
@@ -112,6 +158,7 @@ static void copyout(stream *in)
// eof
break;
}
+   errno = 0;
nwritten = fwrite(buf, 1, nread, out);
if (nwritten != (size_t)nread)
croak(2, "Write error after %lu bytes: %s", total + 
nwritten, strerror(errno));
@@ -119,7 +166,38 @@ static void copyout(stream *in)
}
 
fflush(out);
-   // DO NOT CLOSE out!  It's an alias for stdout.
+   // DO NOT fclose(out)!  It's an alias for stdout.
+}
+
+
+static void copy_from_stdin(stream *out)
+{
+   FILE *in;
+   char buf[1024];
+   size_t nread;
+   ssize_t nwritten;
+   unsigned long total = 0;
+
+   // We can't flush stdin but it hasn't been used yet so with any
+   // luck, no input is buffered yet
+   in = fdopen(fileno(stdin), "rb");
+
+   while (1) {
+   errno = 0;
+   nread = fread(buf, 1, sizeof(buf), in);
+   if (nread == 0) {
+   if (errno != 0)
+   croak(2, "Error reading from stream after %lu 
bytes: %s", total, strerror(errno));
+   else
+   break;
+   }
+   nwritten = mnstr_write(out, buf, 1, nread);
+   if ((size_t)nwritten != nread)
+   croak(2, "Write error after %lu bytes: %s", total + 
nwritten, mnstr_error(out));
+   total += nwritten;
+   }
+
+   // DO NOT fclose(in)!  It's an alias for stdin.
 }
 
 
@@ -140,4 +218,23 @@ opener_rastream(char *filename)
if (s == NULL)
croak(2, "Error opening file '%s': %s", filename, 
strerror(errno));
return s;
+}
+
+static stream *
+opener_wstream(char *filename)
+{
+   stream *s = open_wstream(filename);
+   if (s == NULL)
+   croak(2, "Error opening file '%s': %s", filename, 
strerror(errno));
+   return s;
+}
+
+
+static stream *

MonetDB: makelibstreamgreatagain - Generate testdata on demand

[Joeri van Ruth]

Changeset: 321ccd5d2178 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=321ccd5d2178
Added Files:
common/stream/Tests/testdata.py
Removed Files:
common/stream/Tests/gen_data.py
Modified Files:
common/stream/Tests/streamtests.py
Branch: makelibstreamgreatagain
Log Message:

Generate testdata on demand


diffs (truncated from 426 to 300 lines):

diff --git a/common/stream/Tests/gen_data.py b/common/stream/Tests/gen_data.py
deleted file mode 100755
--- a/common/stream/Tests/gen_data.py
+++ /dev/null
@@ -1,103 +0,0 @@
-#!/usr/bin/env python3
-
-import bz2
-import gzip
-import hashlib
-import json
-import lz4.frame
-import lzma
-import os
-
-import sys
-
-BOM = b'\xEF\xBB\xBF'
-
-# Note: this is a byte string, not a text string.
-INPUT = gzip.open('1661-0.txt.gz', 'rb').read()
-
-# Note: it uses DOS line endings.
-# This is important because the streams library leaves those alone,
-# even in text mode.
-assert INPUT.find(b'\x0d\x0a') >= 0
-
-# Contents of the catalog file.
-DESCRIPTORS = []
-
-def write(name, compression, content):
-filename = 'data/' + name + '.txt'
-if not compression:
-f = open(filename, 'wb')
-elif compression == 'gz':
-filename += '.gz'
-f = gzip.GzipFile(filename, 'wb', mtime=131875200, compresslevel=9)
-elif compression == 'bz2':
-filename += '.bz2'
-f = bz2.BZ2File(filename, 'wb', compresslevel=9)
-elif compression == 'xz':
-filename += '.xz'
-f = lzma.LZMAFile(filename, 'wb', preset=9)
-elif compression == 'lz4':
-filename += '.lz4'
-f = lz4.frame.LZ4FrameFile(filename, 'wb', compression_level=9)
-else:
-raise Exception("Unknown compression scheme: " + compression)
-
-f.write(content)
-f.close()
-
-# DIRTY HACK
-# For the time being the test script normalizes everything to unix
-# line endings before doing any comparisons.
-#
-# Here we make sure the reference data matches that
-content = content.replace(b'\r\n', b'\n')
-
-has_bom = content.startswith(BOM)
-without_bom = content[3:] if has_bom else content
-
-n = 5
-descr = dict(
-filename=os.path.basename(filename),
-has_bom = has_bom,
-size_without_bom = len(without_bom),
-md5 = hashlib.md5(content).hexdigest(),
-md5_without_bom = hashlib.md5(without_bom).hexdigest(),
-start_without_bom = str(without_bom[:n], 'ascii'),
-end_without_bom = str(without_bom[-n:], 'ascii'),
-)
-
-DESCRIPTORS.append(descr)
-
-
-def write_all_compressions(name, content, limit):
-if limit:
-content = content[:limit]
-write(name, None, content)
-write(name, 'gz', content)
-write(name, 'bz2', content)
-write(name, 'xz', content)
-write(name, 'lz4', content)
-
-def write_all(name, content, limit=None):
-write_all_compressions(name, content, limit)
-write_all_compressions(name + '_bom', BOM + content, limit)
-
-
-# Whole file
-write_all('sherlock', INPUT)
-
-# Empty file
-write_all('empty', b'')
-
-# First 16 lines
-head = b'\n'.join(INPUT.split(b'\n')[:16]) + b'\n'
-write_all('small', head)
-
-# Buffer size boundary cases
-for base_size in [1024, 2048, 4096, 8192, 16384]:
-for delta in [-1, 0, 1]:
-size = base_size + delta
-write_all(f'block{size}', INPUT, size)
-
-# Write the descriptor
-json.dump(DESCRIPTORS, open('data/testcases.json', 'w'), indent=4)
diff --git a/common/stream/Tests/streamtests.py 
b/common/stream/Tests/streamtests.py
--- a/common/stream/Tests/streamtests.py
+++ b/common/stream/Tests/streamtests.py
@@ -1,94 +1,57 @@
 #!/usr/bin/env python3
 
+import testdata
+
 import hashlib
 import json
 import os
 import subprocess
 import sys
 
-BASE = os.environ.get('TSTSRCDIR', 
os.path.dirname(os.path.abspath(sys.argv[0])))
-DATA = os.path.join(BASE, 'data')
 
 BOM = b'\xEF\xBB\xBF'
 
 
-class Doc:
-   __slots__ = ['filename', 'has_bom', 'size_without_bom', 'md5',
-   'md5_without_bom', 'start_without_bom', 'end_without_bom']
-
-   def __init__(self, datadict):
-   self.filename = datadict['filename']
-   self.has_bom = datadict['has_bom']
-   self.size_without_bom = datadict['size_without_bom']
-   self.md5 = datadict['md5']
-   self.md5_without_bom = datadict['md5_without_bom']
-   self.start_without_bom = bytes(datadict['start_without_bom'], 
'utf-8')
-   self.end_without_bom = bytes(datadict['end_without_bom'], 
'utf-8')
-
-   def verify(self, text, text_mode):
-   # DIRTY HACK
-   # For the time being we completely ignore the line ending issue.
-   # The reason is that reading a DOS file on Unix correctly yields
-   # \r\n's. Because Unix does not do line ending translation.
-   # However, the compression algorithms DO translate newlines
-   # 

MonetDB: makelibstreamgreatagain - Integrate in Mtest.py infra

[Joeri van Ruth]

Changeset: 02f08325e908 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=02f08325e908
Added Files:
common/stream/Tests/read_bz2.stable.err
common/stream/Tests/read_bz2.stable.out
common/stream/Tests/read_gz.stable.err
common/stream/Tests/read_gz.stable.out
common/stream/Tests/read_lz4.stable.err
common/stream/Tests/read_lz4.stable.out
common/stream/Tests/read_uncompressed.stable.err
common/stream/Tests/read_uncompressed.stable.out
common/stream/Tests/read_xz.stable.err
common/stream/Tests/read_xz.stable.out
Branch: makelibstreamgreatagain
Log Message:

Integrate in Mtest.py infra


diffs (truncated from 890 to 300 lines):

diff --git a/common/stream/Tests/read_bz2.stable.err 
b/common/stream/Tests/read_bz2.stable.err
new file mode 100644
--- /dev/null
+++ b/common/stream/Tests/read_bz2.stable.err
@@ -0,0 +1,12 @@
+stderr of test 'read_bz2` in directory 'common/stream` itself:
+
+
+# 14:14:04 >  
+# 14:14:04 >  "/usr/bin/python3" "read_bz2.py" "read_bz2"
+# 14:14:04 >  
+
+
+# 14:14:05 >  
+# 14:14:05 >  "Done."
+# 14:14:05 >  
+
diff --git a/common/stream/Tests/read_bz2.stable.out 
b/common/stream/Tests/read_bz2.stable.out
new file mode 100644
--- /dev/null
+++ b/common/stream/Tests/read_bz2.stable.out
@@ -0,0 +1,156 @@
+stdout of test 'read_bz2` in directory 'common/stream` itself:
+
+
+# 14:14:04 >  
+# 14:14:04 >  "/usr/bin/python3" "read_bz2.py" "read_bz2"
+# 14:14:04 >  
+
+
+Test read rstream sherlock.txt.bz2 OK
+
+Test read rastream sherlock.txt.bz2 OK
+
+Test read rstream sherlock_bom.txt.bz2 OK
+
+Test read rastream sherlock_bom.txt.bz2 OK
+
+Test read rstream empty.txt.bz2 OK
+
+Test read rastream empty.txt.bz2 OK
+
+Test read rstream empty_bom.txt.bz2 OK
+
+Test read rastream empty_bom.txt.bz2 OK
+
+Test read rstream small.txt.bz2 OK
+
+Test read rastream small.txt.bz2 OK
+
+Test read rstream small_bom.txt.bz2 OK
+
+Test read rastream small_bom.txt.bz2 OK
+
+Test read rstream block1023.txt.bz2 OK
+
+Test read rastream block1023.txt.bz2 OK
+
+Test read rstream block1023_bom.txt.bz2 OK
+
+Test read rastream block1023_bom.txt.bz2 OK
+
+Test read rstream block1024.txt.bz2 OK
+
+Test read rastream block1024.txt.bz2 OK
+
+Test read rstream block1024_bom.txt.bz2 OK
+
+Test read rastream block1024_bom.txt.bz2 OK
+
+Test read rstream block1025.txt.bz2 OK
+
+Test read rastream block1025.txt.bz2 OK
+
+Test read rstream block1025_bom.txt.bz2 OK
+
+Test read rastream block1025_bom.txt.bz2 OK
+
+Test read rstream block2047.txt.bz2 OK
+
+Test read rastream block2047.txt.bz2 OK
+
+Test read rstream block2047_bom.txt.bz2 OK
+
+Test read rastream block2047_bom.txt.bz2 OK
+
+Test read rstream block2048.txt.bz2 OK
+
+Test read rastream block2048.txt.bz2 OK
+
+Test read rstream block2048_bom.txt.bz2 OK
+
+Test read rastream block2048_bom.txt.bz2 OK
+
+Test read rstream block2049.txt.bz2 OK
+
+Test read rastream block2049.txt.bz2 OK
+
+Test read rstream block2049_bom.txt.bz2 OK
+
+Test read rastream block2049_bom.txt.bz2 OK
+
+Test read rstream block4095.txt.bz2 OK
+
+Test read rastream block4095.txt.bz2 OK
+
+Test read rstream block4095_bom.txt.bz2 OK
+
+Test read rastream block4095_bom.txt.bz2 OK
+
+Test read rstream block4096.txt.bz2 OK
+
+Test read rastream block4096.txt.bz2 OK
+
+Test read rstream block4096_bom.txt.bz2 OK
+
+Test read rastream block4096_bom.txt.bz2 OK
+
+Test read rstream block4097.txt.bz2 OK
+
+Test read rastream block4097.txt.bz2 OK
+
+Test read rstream block4097_bom.txt.bz2 OK
+
+Test read rastream block4097_bom.txt.bz2 OK
+
+Test read rstream block8191.txt.bz2 OK
+
+Test read rastream block8191.txt.bz2 OK
+
+Test read rstream block8191_bom.txt.bz2 OK
+
+Test read rastream block8191_bom.txt.bz2 OK
+
+Test read rstream block8192.txt.bz2 OK
+
+Test read rastream block8192.txt.bz2 OK
+
+Test read rstream block8192_bom.txt.bz2 OK
+
+Test read rastream block8192_bom.txt.bz2 OK
+
+Test read rstream block8193.txt.bz2 OK
+
+Test read rastream block8193.txt.bz2 OK
+
+Test read rstream block8193_bom.txt.bz2 OK
+
+Test read rastream block8193_bom.txt.bz2 OK
+
+Test read rstream block16383.txt.bz2 OK
+
+Test read rastream block16383.txt.bz2 OK
+
+Test read rstream block16383_bom.txt.bz2 OK
+
+Test read rastream block16383_bom.txt.bz2 OK
+
+Test read rstream block16384.txt.bz2 OK
+
+Test read rastream block16384.txt.bz2 OK
+
+Test read rstream block16384_bom.txt.bz2 OK
+
+Test read rastream block16384_bom.txt.bz2 OK
+
+Test read rstream block16385.txt.bz2 OK
+
+Test read rastream block16385.txt.bz2 OK
+
+Test read rstream block16385_bom.txt.bz2 OK
+
+Test read rastream block16385_bom.txt.bz2 OK
+
+# 14:14:05 >  
+# 14:14:05 >  "Done."
+# 14:14:05 >  
+
diff --git a/common/stream/Tests/read_gz.stable.err 
b/common/stream/Tests/read_gz.stable.err
new file mode 100644
--- /dev/null
+++ b/common/stream/Tests/read_gz.stable.err
@@ -0,0 +1,12 @@
+stderr of test 'read_gz` in directory 'common/stream` itself:
+

MonetDB: makelibstreamgreatagain - Make 1024 byte testfile truly...

[Joeri van Ruth]

Changeset: 8433cf8a53f2 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8433cf8a53f2
Modified Files:
common/stream/Tests/gen_data.py
common/stream/Tests/streamtests.py
Branch: makelibstreamgreatagain
Log Message:

Make 1024 byte testfile truly 1024B, even with BOM


diffs (59 lines):

diff --git a/common/stream/Tests/gen_data.py b/common/stream/Tests/gen_data.py
--- a/common/stream/Tests/gen_data.py
+++ b/common/stream/Tests/gen_data.py
@@ -55,7 +55,7 @@ def write(name, compression, content):
 has_bom = content.startswith(BOM)
 without_bom = content[3:] if has_bom else content
 
-n = 8
+n = 5
 descr = dict(
 filename=os.path.basename(filename),
 has_bom = has_bom,
@@ -69,16 +69,18 @@ def write(name, compression, content):
 DESCRIPTORS.append(descr)
 
 
-def write_all_compressions(name, content):
+def write_all_compressions(name, content, limit):
+if limit:
+content = content[:limit]
 write(name, None, content)
 write(name, 'gz', content)
 write(name, 'bz2', content)
 write(name, 'xz', content)
 write(name, 'lz4', content)
 
-def write_all(name, content):
-write_all_compressions(name, content)
-write_all_compressions(name + '_bom', BOM + content)
+def write_all(name, content, limit=None):
+write_all_compressions(name, content, limit)
+write_all_compressions(name + '_bom', BOM + content, limit)
 
 
 # Whole file
@@ -95,7 +97,7 @@ write_all('small', head)
 for base_size in [1024, 2048, 4096, 8192, 16384]:
 for delta in [-1, 0, 1]:
 size = base_size + delta
-write_all(f'block{size}', INPUT[:size])
+write_all(f'block{size}', INPUT, size)
 
 # Write the descriptor
 json.dump(DESCRIPTORS, open('data/testcases.json', 'w'), indent=4)
diff --git a/common/stream/Tests/streamtests.py 
b/common/stream/Tests/streamtests.py
--- a/common/stream/Tests/streamtests.py
+++ b/common/stream/Tests/streamtests.py
@@ -66,9 +66,8 @@ class Doc:
if digest != self.md5_without_bom:
return "MD5 digest mismatch: generated output differs."
 
-   n = 8
-   start = text[:n]
-   end = text[-n:]
+   start = text[:len(self.start_without_bom)]
+   end = text[-len(self.end_without_bom):]
if start != self.start_without_bom:
return f"Expected text to start with 
{repr(self.start_without_bom)}, found {repr(start)}"
if end != self.end_without_bom:
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Do not interpret a short read...

[Joeri van Ruth]

Changeset: e94379f9eac6 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e94379f9eac6
Modified Files:
clients/examples/C/streamcat.c
Branch: makelibstreamgreatagain
Log Message:

Do not interpret a short read as EOF


diffs (23 lines):

diff --git a/clients/examples/C/streamcat.c b/clients/examples/C/streamcat.c
--- a/clients/examples/C/streamcat.c
+++ b/clients/examples/C/streamcat.c
@@ -104,14 +104,14 @@ static void copyout(stream *in)
fflush(stdout);
out = fdopen(fileno(stdout), "wb");
 
-   bool done_looping = false;
-   while (!done_looping) {
+   while (1) {
nread = mnstr_read(in, buf, 1, sizeof(buf));
if (nread < 0)
croak(2, "Error reading from stream after %lu bytes: 
%s", total, mnstr_error(in));
-   if (nread < (ssize_t)sizeof(buf))
-   done_looping = true;
-
+   if (nread == 0) {
+   // eof
+   break;
+   }
nwritten = fwrite(buf, 1, nread, out);
if (nwritten != (size_t)nread)
croak(2, "Write error after %lu bytes: %s", total + 
nwritten, strerror(errno));
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - In the libstream test, ignore...

[Joeri van Ruth]

Changeset: 2dbe7c4cd714 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=2dbe7c4cd714
Modified Files:
common/stream/Tests/gen_data.py
common/stream/Tests/streamtests.py
Branch: makelibstreamgreatagain
Log Message:

In the libstream test, ignore line endings for the moment

They are inconsistent between uncompressed and compressed reading
when using open_rastream().


diffs (88 lines):

diff --git a/common/stream/Tests/gen_data.py b/common/stream/Tests/gen_data.py
--- a/common/stream/Tests/gen_data.py
+++ b/common/stream/Tests/gen_data.py
@@ -45,6 +45,13 @@ def write(name, compression, content):
 f.write(content)
 f.close()
 
+# DIRTY HACK
+# For the time being the test script normalizes everything to unix
+# line endings before doing any comparisons.
+#
+# Here we make sure the reference data matches that
+content = content.replace(b'\r\n', b'\n')
+
 has_bom = content.startswith(BOM)
 without_bom = content[3:] if has_bom else content
 
diff --git a/common/stream/Tests/streamtests.py 
b/common/stream/Tests/streamtests.py
--- a/common/stream/Tests/streamtests.py
+++ b/common/stream/Tests/streamtests.py
@@ -25,25 +25,36 @@ class Doc:
self.start_without_bom = bytes(datadict['start_without_bom'], 
'utf-8')
self.end_without_bom = bytes(datadict['end_without_bom'], 
'utf-8')
 
-   def verify(self, text, expect_bom_stripped):
+   def verify(self, text, text_mode):
+   # DIRTY HACK
+   # For the time being we completely ignore the line ending issue.
+   # The reason is that reading a DOS file on Unix correctly yields
+   # \r\n's. Because Unix does not do line ending translation.
+   # However, the compression algorithms DO translate newlines
+   # when reading in text mode, even on Unix.
+   #
+   # Until we figure out how we want to deal with that we pretend
+   # the issue doesn't exist.
+   text = text.replace(b'\r\n', b'\n')
+
bom_found = text.startswith(BOM)
 
-   # | HAS_BOM | EXPECT_BOM_REMOVED | BOM_FOUND | RESULT   
|
-   # 
|-++---+--|
-   # | False   | *  | False | OK   
|
-   # | False   | *  | True  | Somehow, a BOM 
was inserted! |
-   # | True| False  | False | The BOM should 
not have been removed |
-   # | True| False  | True  | OK   
|
-   # | True| True   | False | OK   
|
-   # | True| True   | True  | The BOM should 
have been removed |
+   # | HAS_BOM | TEXT_MODE | BOM_FOUND | RESULT
   |
+   # 
|-+---+---+--|
+   # | False   | * | False | OK
   |
+   # | False   | * | True  | Somehow, a BOM was 
inserted! |
+   # | True| False | False | The BOM should not have 
been removed |
+   # | True| False | True  | OK
   |
+   # | True| True  | False | OK
   |
+   # | True| True  | True  | The BOM should have been 
removed |
if not self.has_bom:
if bom_found:
return "Somehow, a BOM was inserted!"
if self.has_bom:
-   if expect_bom_stripped and bom_found:
-   return "The BOM should have been removed"
-   if not expect_bom_stripped and not bom_found:
-   return "The BOM should not have been removed"
+   if text_mode and bom_found:
+   return "In text mode, the BOM should have been 
removed"
+   if not text_mode and not bom_found:
+   return "In binary mode, the BOM should not have 
been removed"
 
if bom_found:
text = text[len(BOM):]
@@ -73,7 +84,7 @@ def get_docs():
raw = json.load(open(os.path.join(DATA, 'testcases.json')))
return [Doc(r) for r in raw]
 
-def test_read(opener, strips_bom, doc):
+def test_read(opener, text_mode, doc):
test = f"read {opener} {doc.filename}"
 
print()
@@ -85,7 +96,7 @@ def test_read(opener, strips_bom, doc):
return False
 
output = results.stdout or b""
-   complaint = 

MonetDB: makelibstreamgreatagain - Try to ensure binary stdout o...

[Joeri van Ruth]

Changeset: 39c92336b9ca for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=39c92336b9ca
Modified Files:
clients/examples/C/streamcat.c
Branch: makelibstreamgreatagain
Log Message:

Try to ensure binary stdout on Windows

(untested)


diffs (56 lines):

diff --git a/clients/examples/C/streamcat.c b/clients/examples/C/streamcat.c
--- a/clients/examples/C/streamcat.c
+++ b/clients/examples/C/streamcat.c
@@ -22,7 +22,7 @@ typedef stream *(*opener_fun)(char *file
 static stream *opener_rstream(char *filename);
 static stream *opener_rastream(char *filename);
 
-static void copyout(stream *s, FILE *out);
+static void copyout(stream *in);
 
 _Noreturn static void croak(int status, const char *msg, ...)
__attribute__((__format__(__printf__, 2, 3)));
@@ -86,24 +86,29 @@ int cmd_read(char *argv[])
 
s = opener(filename);
 
-   copyout(s, stdout);
+   copyout(s);
 
return 0;
 }
 
 
-static void copyout(stream *s, FILE *out)
+static void copyout(stream *in)
 {
+   FILE *out;
char buf[1024];
ssize_t nread;
size_t nwritten;
unsigned long total = 0;
 
+   // Try to get binary stdout on Windows
+   fflush(stdout);
+   out = fdopen(fileno(stdout), "wb");
+
bool done_looping = false;
while (!done_looping) {
-   nread = mnstr_read(s, buf, 1, sizeof(buf));
+   nread = mnstr_read(in, buf, 1, sizeof(buf));
if (nread < 0)
-   croak(2, "Error reading from stream after %lu bytes: 
%s", total, mnstr_error(s));
+   croak(2, "Error reading from stream after %lu bytes: 
%s", total, mnstr_error(in));
if (nread < (ssize_t)sizeof(buf))
done_looping = true;
 
@@ -112,7 +117,9 @@ static void copyout(stream *s, FILE *out
croak(2, "Write error after %lu bytes: %s", total + 
nwritten, strerror(errno));
total += nwritten;
}
+
fflush(out);
+   // DO NOT CLOSE out!  It's an alias for stdout.
 }
 
 
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Refactor for easier debugging

[Joeri van Ruth]

Changeset: 822e0754d570 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=822e0754d570
Modified Files:
common/stream/Tests/read_bz2.py
common/stream/Tests/read_gz.py
common/stream/Tests/read_lz4.py
common/stream/Tests/read_uncompressed.py
common/stream/Tests/read_xz.py
common/stream/Tests/streamtests.py
Branch: makelibstreamgreatagain
Log Message:

Refactor for easier debugging


diffs (140 lines):

diff --git a/common/stream/Tests/read_bz2.py b/common/stream/Tests/read_bz2.py
--- a/common/stream/Tests/read_bz2.py
+++ b/common/stream/Tests/read_bz2.py
@@ -5,5 +5,5 @@ import sys
 
 filter = lambda f: f.endswith('.txt.bz2')
 
-if not streamtests.all_read_tests(filter):
+if streamtests.all_read_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/read_gz.py b/common/stream/Tests/read_gz.py
--- a/common/stream/Tests/read_gz.py
+++ b/common/stream/Tests/read_gz.py
@@ -5,5 +5,5 @@ import sys
 
 filter = lambda f: f.endswith('.txt.gz')
 
-if not streamtests.all_read_tests(filter):
+if streamtests.all_read_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/read_lz4.py b/common/stream/Tests/read_lz4.py
--- a/common/stream/Tests/read_lz4.py
+++ b/common/stream/Tests/read_lz4.py
@@ -5,5 +5,5 @@ import sys
 
 filter = lambda f: f.endswith('.txt.lz4')
 
-if not streamtests.all_read_tests(filter):
+if streamtests.all_read_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/read_uncompressed.py 
b/common/stream/Tests/read_uncompressed.py
--- a/common/stream/Tests/read_uncompressed.py
+++ b/common/stream/Tests/read_uncompressed.py
@@ -5,5 +5,5 @@ import sys
 
 filter = lambda f: f.endswith('.txt')
 
-if not streamtests.all_read_tests(filter):
+if streamtests.all_read_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/read_xz.py b/common/stream/Tests/read_xz.py
--- a/common/stream/Tests/read_xz.py
+++ b/common/stream/Tests/read_xz.py
@@ -5,5 +5,5 @@ import sys
 
 filter = lambda f: f.endswith('.txt.xz')
 
-if not streamtests.all_read_tests(filter):
+if streamtests.all_read_tests(filter) != 0:
sys.exit(1)
diff --git a/common/stream/Tests/streamtests.py 
b/common/stream/Tests/streamtests.py
--- a/common/stream/Tests/streamtests.py
+++ b/common/stream/Tests/streamtests.py
@@ -6,7 +6,7 @@ import os
 import subprocess
 import sys
 
-BASE = os.environ.get('TSTSRCDIR', os.getcwd())
+BASE = os.environ.get('TSTSRCDIR', 
os.path.dirname(os.path.abspath(sys.argv[0])))
 DATA = os.path.join(BASE, 'data')
 
 BOM = b'\xEF\xBB\xBF'
@@ -95,48 +95,42 @@ def test_read(opener, strips_bom, doc):
return True
 
 
+def test_reads(doc):
+   failures = 0
+
+   # rstream does not strip BOM
+   failures += not test_read('rstream', False, doc)
+
+   # rastream does strip the BOM
+   failures += not test_read('rastream', True, doc)
+
+   return failures
+
+
 def all_read_tests(filename_filter):
-   ok = True
+   failures = 0
for d in get_docs():
if not filename_filter(d.filename):
continue
 
# rstream does not strip BOM
-   ok = ok & test_read('rstream', False, d)
-
-   # rastream does strip the BOM
-   ok = ok & test_read('rastream', True, d)
-
-   return ok
-
-# import os
-# import subprocess
-# import sys
-
-# # Used by InputFile.path()
+   failures += test_reads(d)
 
-# class InputFile:
-#  __slots__ = ['filename', 'has_bom', 'size', 'md5sum', 'first100bytes', 
'last100bytes']
-
-#  def __init__(self, filename, has_bom, size, md5sum, first100bytes, 
last100bytes):
-#  self.filename = filename
-#  self.has_bom = has_bom
-#  self.size = size
-#  self.md5sum = md5sum
-#  self.first100bytes = first100bytes
-#  self.last100bytes = last100bytes
-
-#  def path(self, compression):
-#  extension = '.' + compression if compression else ''
-#  return os.path.join(BASE, 'data', self.filename + extension)
-
-#  def verify(self, result_bytes, ):
-#  pass
+   return failures
 
 
-# def main():
-#  pass
+def main(kind, file):
+   failures = 0
+   if kind == "read":
+   docs = [d for d in get_docs() if d.filename == file]
+   print(f"Found {len(docs)} matches.")
+   print()
+   for d in docs:
+   failures += test_reads(d)
 
+   return failures == 0
 
-# if __name__ == "__main__":
-# sys.exit(main() or 0)
+if __name__ == "__main__":
+   docname = "small.txt.gz"
+   doc = [d for d in get_docs() if d.filename == docname][0]
+   test_read('rastream', True, doc)
\ No newline at end of file
___
checkin-list mailing list
checkin-list@monetdb.org

MonetDB: makelibstreamgreatagain - Properly register as mtests

[Joeri van Ruth]

Changeset: e1fd006b1c46 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e1fd006b1c46
Modified Files:
common/stream/Tests/All
Branch: makelibstreamgreatagain
Log Message:

Properly register as mtests


diffs (11 lines):

diff --git a/common/stream/Tests/All b/common/stream/Tests/All
--- a/common/stream/Tests/All
+++ b/common/stream/Tests/All
@@ -1,1 +1,5 @@
-read_uncompressed
\ No newline at end of file
+read_uncompressed
+read_bz2
+read_gz
+read_lz4
+read_xz
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Add lz4 support to stream rea...

[Joeri van Ruth]

Changeset: 0bd1bcbe8037 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0bd1bcbe8037
Added Files:
common/stream/Tests/read_lz4.py
Modified Files:
common/stream/Tests/gen_data.py
Branch: makelibstreamgreatagain
Log Message:

Add lz4 support to stream read tests


diffs (43 lines):

diff --git a/common/stream/Tests/gen_data.py b/common/stream/Tests/gen_data.py
--- a/common/stream/Tests/gen_data.py
+++ b/common/stream/Tests/gen_data.py
@@ -4,6 +4,7 @@ import bz2
 import gzip
 import hashlib
 import json
+import lz4.frame
 import lzma
 import os
 
@@ -35,6 +36,9 @@ def write(name, compression, content):
 elif compression == 'xz':
 filename += '.xz'
 f = lzma.LZMAFile(filename, 'wb', preset=9)
+elif compression == 'lz4':
+filename += '.lz4'
+f = lz4.frame.LZ4FrameFile(filename, 'wb', compression_level=9)
 else:
 raise Exception("Unknown compression scheme: " + compression)
 
@@ -63,6 +67,7 @@ def write_all_compressions(name, content
 write(name, 'gz', content)
 write(name, 'bz2', content)
 write(name, 'xz', content)
+write(name, 'lz4', content)
 
 def write_all(name, content):
 write_all_compressions(name, content)
diff --git a/common/stream/Tests/read_lz4.py b/common/stream/Tests/read_lz4.py
new file mode 100755
--- /dev/null
+++ b/common/stream/Tests/read_lz4.py
@@ -0,0 +1,9 @@
+#!/usr/bin/env python3
+
+import streamtests
+import sys
+
+filter = lambda f: f.endswith('.txt.lz4')
+
+if not streamtests.all_read_tests(filter):
+   sys.exit(1)
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Add xz support to stream read...

[Joeri van Ruth]

Changeset: 46c78c31cb96 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=46c78c31cb96
Added Files:
common/stream/Tests/read_xz.py
Modified Files:
common/stream/Tests/gen_data.py
Branch: makelibstreamgreatagain
Log Message:

Add xz support to stream read tests


diffs (44 lines):

diff --git a/common/stream/Tests/gen_data.py b/common/stream/Tests/gen_data.py
--- a/common/stream/Tests/gen_data.py
+++ b/common/stream/Tests/gen_data.py
@@ -4,6 +4,7 @@ import bz2
 import gzip
 import hashlib
 import json
+import lzma
 import os
 
 import sys
@@ -31,6 +32,9 @@ def write(name, compression, content):
 elif compression == 'bz2':
 filename += '.bz2'
 f = bz2.BZ2File(filename, 'wb', compresslevel=9)
+elif compression == 'xz':
+filename += '.xz'
+f = lzma.LZMAFile(filename, 'wb', preset=9)
 else:
 raise Exception("Unknown compression scheme: " + compression)
 
@@ -58,7 +62,7 @@ def write_all_compressions(name, content
 write(name, None, content)
 write(name, 'gz', content)
 write(name, 'bz2', content)
-
+write(name, 'xz', content)
 
 def write_all(name, content):
 write_all_compressions(name, content)
diff --git a/common/stream/Tests/read_xz.py b/common/stream/Tests/read_xz.py
new file mode 100755
--- /dev/null
+++ b/common/stream/Tests/read_xz.py
@@ -0,0 +1,9 @@
+#!/usr/bin/env python3
+
+import streamtests
+import sys
+
+filter = lambda f: f.endswith('.txt.xz')
+
+if not streamtests.all_read_tests(filter):
+   sys.exit(1)
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Fixup: set compress level to ...

[Joeri van Ruth]

Changeset: 9f4bde48298b for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9f4bde48298b
Modified Files:
common/stream/Tests/gen_data.py
Branch: makelibstreamgreatagain
Log Message:

Fixup: set compress level to 9 explicitly


diffs (16 lines):

diff --git a/common/stream/Tests/gen_data.py b/common/stream/Tests/gen_data.py
--- a/common/stream/Tests/gen_data.py
+++ b/common/stream/Tests/gen_data.py
@@ -27,10 +27,10 @@ def write(name, compression, content):
 f = open(filename, 'wb')
 elif compression == 'gz':
 filename += '.gz'
-f = gzip.GzipFile(filename, 'wb', mtime=131875200)
+f = gzip.GzipFile(filename, 'wb', mtime=131875200, compresslevel=9)
 elif compression == 'bz2':
 filename += '.bz2'
-f = bz2.BZ2File(filename, 'wb')
+f = bz2.BZ2File(filename, 'wb', compresslevel=9)
 else:
 raise Exception("Unknown compression scheme: " + compression)
 
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Fixup message from streamtest...

[Joeri van Ruth]

Changeset: 8e1d75fe105f for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8e1d75fe105f
Modified Files:
common/stream/Tests/streamtests.py
Branch: makelibstreamgreatagain
Log Message:

Fixup message from streamtests.py


diffs (13 lines):

diff --git a/common/stream/Tests/streamtests.py 
b/common/stream/Tests/streamtests.py
--- a/common/stream/Tests/streamtests.py
+++ b/common/stream/Tests/streamtests.py
@@ -49,8 +49,7 @@ class Doc:
text = text[len(BOM):]
 
if len(text) != self.size_without_bom:
-   return "Size mismatch"
-   # return f"Size mismatch: found {len(text)} bytes, 
expected {self.size_without_bom}"
+   return f"Size mismatch: found {len(text)} bytes, 
expected {self.size_without_bom}"
 
digest = hashlib.md5(text).hexdigest()
if digest != self.md5_without_bom:
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Add gzip support to stream re...

[Joeri van Ruth]

Changeset: e1e89d54f03f for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e1e89d54f03f
Added Files:
common/stream/Tests/read_gz.py
Branch: makelibstreamgreatagain
Log Message:

Add gzip support to stream read tests


diffs (14 lines):

diff --git a/common/stream/Tests/read_gz.py b/common/stream/Tests/read_gz.py
new file mode 100755
--- /dev/null
+++ b/common/stream/Tests/read_gz.py
@@ -0,0 +1,9 @@
+#!/usr/bin/env python3
+
+import streamtests
+import sys
+
+filter = lambda f: f.endswith('.txt.gz')
+
+if not streamtests.all_read_tests(filter):
+   sys.exit(1)
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Add bz2 support to stream rea...

[Joeri van Ruth]

Changeset: 38aa2b098b7a for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=38aa2b098b7a
Added Files:
common/stream/Tests/read_bz2.py
Modified Files:
common/stream/Tests/gen_data.py
Branch: makelibstreamgreatagain
Log Message:

Add bz2 support to stream read tests


diffs (42 lines):

diff --git a/common/stream/Tests/gen_data.py b/common/stream/Tests/gen_data.py
--- a/common/stream/Tests/gen_data.py
+++ b/common/stream/Tests/gen_data.py
@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 
+import bz2
 import gzip
 import hashlib
 import json
@@ -27,6 +28,9 @@ def write(name, compression, content):
 elif compression == 'gz':
 filename += '.gz'
 f = gzip.GzipFile(filename, 'wb', mtime=131875200)
+elif compression == 'bz2':
+filename += '.bz2'
+f = bz2.BZ2File(filename, 'wb')
 else:
 raise Exception("Unknown compression scheme: " + compression)
 
@@ -53,6 +57,7 @@ def write(name, compression, content):
 def write_all_compressions(name, content):
 write(name, None, content)
 write(name, 'gz', content)
+write(name, 'bz2', content)
 
 
 def write_all(name, content):
diff --git a/common/stream/Tests/read_bz2.py b/common/stream/Tests/read_bz2.py
new file mode 100755
--- /dev/null
+++ b/common/stream/Tests/read_bz2.py
@@ -0,0 +1,9 @@
+#!/usr/bin/env python3
+
+import streamtests
+import sys
+
+filter = lambda f: f.endswith('.txt.bz2')
+
+if not streamtests.all_read_tests(filter):
+   sys.exit(1)
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: makelibstreamgreatagain - Add tests for reading files w...

[Joeri van Ruth]

Changeset: 1c9f52c14249 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1c9f52c14249
Added Files:
clients/examples/C/streamcat.c
common/stream/Tests/1661-0.txt.gz
common/stream/Tests/All
common/stream/Tests/gen_data.py
common/stream/Tests/read_uncompressed.py
common/stream/Tests/streamtests.py
Modified Files:
clients/examples/C/Makefile.ag
Branch: makelibstreamgreatagain
Log Message:

Add tests for reading files with libstream

The tests invoke clients/examples/C/streamcat.c
to exercise the library.


diffs (truncated from 4794 to 300 lines):

diff --git a/clients/examples/C/Makefile.ag b/clients/examples/C/Makefile.ag
--- a/clients/examples/C/Makefile.ag
+++ b/clients/examples/C/Makefile.ag
@@ -5,13 +5,14 @@
 # Copyright 1997 - July 2008 CWI, August 2008 - 2020 MonetDB B.V.
 
 MTSAFE
-INCLUDES = ../../mapilib
+INCLUDES = ../../mapilib ../../../common/stream
 
 BINS = {
CONDINST = HAVE_TESTING
DIR = bindir
SOURCES = sample0.c sample1.c sample4.c \
-   smack00.c smack01.c
+   smack00.c smack01.c \
+   streamcat.c
LIBS = ../../mapilib/libmapi \
../../../common/stream/libstream \
$(SOCKET_LIBS) $(zlib_LIBS) $(bzip2_LIBS) $(snappy_LIBS) 
$(lz4_LIBS) $(liblzma_LIBS) $(curl_LIBS) $(LTLIBICONV) $(openssl_LIBS)
diff --git a/clients/examples/C/streamcat.c b/clients/examples/C/streamcat.c
new file mode 100644
--- /dev/null
+++ b/clients/examples/C/streamcat.c
@@ -0,0 +1,136 @@
+#include 
+#include 
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+
+const char *USAGE =
+   "Usage:\n"
+   "streamcat read FILENAME OPENER\n"
+   "With OPENER:\n"
+   "- rstream   stream = open_rstream(filename)\n"
+   "- rastream  stream = open_rastream(filename)\n"
+   ;
+
+
+static int cmd_read(char *argv[]);
+typedef stream *(*opener_fun)(char *filename);
+static stream *opener_rstream(char *filename);
+static stream *opener_rastream(char *filename);
+
+static void copyout(stream *s, FILE *out);
+
+_Noreturn static void croak(int status, const char *msg, ...)
+   __attribute__((__format__(__printf__, 2, 3)));
+
+/* Format the message and write it to stderr. Then exit with the given status.
+ * If status is 1, include USAGE in the message.
+ */
+static void
+croak(int status, const char *msg, ...)
+{
+   va_list ap;
+
+   va_start(ap, msg);
+   if (msg != NULL) {
+   fprintf(stderr, "Error: ");
+   vfprintf(stderr, msg, ap);
+   fprintf(stderr, "\n");
+   }
+   va_end(ap);
+   if (status == 1)
+   fprintf(stderr, "%s", USAGE);
+   exit(status);
+}
+
+
+int
+main(int argc, char *argv[])
+{
+   if (argc < 2)
+   croak(1, NULL);
+
+   if (strcmp(argv[1], "read") == 0)
+   return cmd_read(argv+1);
+   else
+   croak(1, "Unknown subcommand '%s'", argv[1]);
+}
+
+
+int cmd_read(char *argv[])
+{
+   char **arg = [1];
+   char *filename = NULL;
+   char *opener_name = NULL;
+   opener_fun opener;
+
+   stream *s = NULL;
+
+   filename = *arg++;
+   if (filename == NULL)
+   croak(1, "Missing filename");
+
+   opener_name = *arg++;
+   if (opener_name == NULL)
+   croak(1, "Missing opener");
+   else if (strcmp(opener_name, "rstream") == 0)
+   opener = opener_rstream;
+   else if (strcmp(opener_name, "rastream") == 0)
+   opener = opener_rastream;
+   else
+   croak(1, "Unknown opener '%s'", opener_name);
+
+   s = opener(filename);
+
+   copyout(s, stdout);
+
+   return 0;
+}
+
+
+static void copyout(stream *s, FILE *out)
+{
+   char buf[1024];
+   ssize_t nread;
+   size_t nwritten;
+   unsigned long total = 0;
+
+   bool done_looping = false;
+   while (!done_looping) {
+   nread = mnstr_read(s, buf, 1, sizeof(buf));
+   if (nread < 0)
+   croak(2, "Error reading from stream after %lu bytes: 
%s", total, mnstr_error(s));
+   if (nread < (ssize_t)sizeof(buf))
+   done_looping = true;
+
+   nwritten = fwrite(buf, 1, nread, out);
+   if (nwritten != (size_t)nread)
+   croak(2, "Write error after %lu bytes: %s", total + 
nwritten, strerror(errno));
+   total += nwritten;
+   }
+   fflush(out);
+}
+
+
+static stream *
+opener_rstream(char *filename)
+{
+   stream *s = open_rstream(filename);
+   if (s == NULL)
+   croak(2, "Error opening file '%s': %s", filename, 
strerror(errno));
+   return s;
+}
+
+
+static stream *
+opener_rastream(char *filename)
+{
+   stream *s = open_rastream(filename);
+   if (s == NULL)
+   croak(2, "Error opening file '%s': %s", 

MonetDB: makelibstreamgreatagain - Make branch

[Joeri van Ruth]

Changeset: 166577d2856c for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=166577d2856c
Modified Files:
common/stream/Makefile.ag
Branch: makelibstreamgreatagain
Log Message:

Make branch


diffs (12 lines):

diff --git a/common/stream/Makefile.ag b/common/stream/Makefile.ag
--- a/common/stream/Makefile.ag
+++ b/common/stream/Makefile.ag
@@ -4,7 +4,7 @@
 #
 # Copyright 1997 - July 2008 CWI, August 2008 - 2020 MonetDB B.V.
 
-## Process this file with automake to produce Makefile.in
+## Process this file with automake to produce Makefile.in 
 
 MTSAFE
 
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: scoping - Approved output

[Pedro Ferreira]

Changeset: 9678762552ce for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9678762552ce
Added Files:
sql/test/scoping/Tests/ambiguity.stable.err
sql/test/scoping/Tests/ambiguity.stable.out
sql/test/scoping/Tests/schema_disambiguity.stable.err
sql/test/scoping/Tests/schema_disambiguity.stable.out
Modified Files:
sql/test/scoping/Tests/ambiguity.sql
sql/test/scoping/Tests/schema_disambiguity.sql
Branch: scoping
Log Message:

Approved output


diffs (truncated from 412 to 300 lines):

diff --git a/sql/test/scoping/Tests/ambiguity.sql 
b/sql/test/scoping/Tests/ambiguity.sql
--- a/sql/test/scoping/Tests/ambiguity.sql
+++ b/sql/test/scoping/Tests/ambiguity.sql
@@ -1,6 +1,4 @@
 -- This is a simple global variable and aims for all kinds of ambiguous use 
cases
-START TRANSACTION;
-
 DECLARE Gvar string;
 SET Gvar='Gvar';
 
@@ -10,43 +8,51 @@ DECLARE Avar string;
 SET Avar='Avar';
 
 CREATE OR REPLACE FUNCTION foo(i string) RETURNS INT
-BEGIN DECLARE i string; set i ='i'; return i; END;
+BEGIN DECLARE i string; set i ='1'; return i; END; --error, i re-declared
 
 CREATE OR REPLACE FUNCTION foo(gvar string) RETURNS INT
-BEGIN DECLARE i string; set i ='i'; return i; END;
+BEGIN DECLARE i string; set i ='1'; return i; END;
 
 CREATE OR REPLACE FUNCTION foo() RETURNS INT
-BEGIN DECLARE i string; set i ='i'; return i; END;
+BEGIN DECLARE i string; set i ='1'; return i; END;
 SELECT foo();
+-- 1
 
 CREATE OR REPLACE FUNCTION Gvar() RETURNS string
 BEGIN DECLARE i string; set i ='function i'; return i; END;
 SELECT Gvar();
+-- function i
 
 -- now scopes
 CREATE OR REPLACE FUNCTION foo() RETURNS string
-BEGIN DECLARE i string; set i =gvar; return i; END;
+BEGIN DECLARE i string; set i =gvar; return i; END; --error, gvar was declared 
on sys schema, so it doesn't exist here
 SELECT foo();
+-- 1
 
 CREATE OR REPLACE FUNCTION Gvar() RETURNS string
-BEGIN DECLARE i string; set i =gvar; return i; END;
+BEGIN DECLARE i string; set i =gvar; return i; END; --error, gvar was declared 
on sys schema, so it doesn't exist here
 SELECT Gvar();
+-- function i
 
 CREATE OR REPLACE FUNCTION foo() RETURNS string
 BEGIN DECLARE i string; set i = sys.gvar; return i; END;
 SELECT foo();
+-- Gvar
 
 CREATE OR REPLACE FUNCTION Gvar() RETURNS string
 BEGIN DECLARE i string; set i =sys.gvar; return i; END;
 SELECT Gvar();
+-- Gvar
 
 CREATE OR REPLACE FUNCTION foo() RETURNS string
-BEGIN DECLARE i string; set i = A.gvar; return i; END;
+BEGIN DECLARE i string; set i = A.gvar; return i; END; --error, variable 
a.gvar doesn't exist
 SELECT foo();
+-- Gvar
 
 CREATE OR REPLACE FUNCTION Gvar() RETURNS string
-BEGIN DECLARE i string; set i =A.gvar; return i; END;
+BEGIN DECLARE i string; set i =A.gvar; return i; END; --error, variable a.gvar 
doesn't exist
 SELECT Gvar();
+-- Gvar
 
 -- procedures
 CREATE OR REPLACE PROCEDURE foo() 
@@ -71,7 +77,9 @@ BEGIN SET SCHEMA A; set avar = 'avar_ass
 CALL avar();
 
 CREATE OR REPLACE PROCEDURE avar() 
-BEGIN SET SCHEMA A; set sys.avar = 'avar_assigned'; END;
+BEGIN SET SCHEMA A; set sys.avar = 'avar_assigned'; END; --error, variable 
sys.avar doesn't exist
 CALL avar();
 
-ROLLBACK;
+SET SCHEMA "sys";
+
+DROP SCHEMA A CASCADE;
diff --git a/sql/test/scoping/Tests/ambiguity.stable.err 
b/sql/test/scoping/Tests/ambiguity.stable.err
new file mode 100644
--- /dev/null
+++ b/sql/test/scoping/Tests/ambiguity.stable.err
@@ -0,0 +1,42 @@
+stderr of test 'ambiguity` in directory 'sql/test/scoping` itself:
+
+
+# 11:19:26 >  
+# 11:19:26 >  "mclient" "-lsql" "-ftest" "-tnone" "-Eutf-8" "-i" "-e" 
"--host=/var/tmp/mtest-220875" "--port=31764"
+# 11:19:26 >  
+
+MAPI  = (monetdb) /var/tmp/mtest-220875/.s.monetdb.31764
+QUERY = CREATE OR REPLACE FUNCTION foo(i string) RETURNS INT
+BEGIN DECLARE i string; set i ='1'; return i; END; --error, i 
re-declared
+ERROR = !DECLARE: Variable 'i' declared as a parameter
+CODE  = 42000
+MAPI  = (monetdb) /var/tmp/mtest-220875/.s.monetdb.31764
+QUERY = CREATE OR REPLACE FUNCTION foo() RETURNS string
+BEGIN DECLARE i string; set i =gvar; return i; END; --error, gvar was 
declared on sys schema, so it doesn't exist here
+ERROR = !SELECT: identifier 'gvar' unknown
+CODE  = 42000
+MAPI  = (monetdb) /var/tmp/mtest-220875/.s.monetdb.31764
+QUERY = CREATE OR REPLACE FUNCTION Gvar() RETURNS string
+BEGIN DECLARE i string; set i =gvar; return i; END; --error, gvar was 
declared on sys schema, so it doesn't exist here
+ERROR = !SELECT: identifier 'gvar' unknown
+CODE  = 42000
+MAPI  = (monetdb) /var/tmp/mtest-220875/.s.monetdb.31764
+QUERY = CREATE OR REPLACE FUNCTION foo() RETURNS string
+BEGIN DECLARE i string; set i = A.gvar; return i; END; --error, 
variable a.gvar doesn't exist
+ERROR = !SELECT: no such column 'a.gvar'
+CODE  = 42S22
+MAPI  = (monetdb) /var/tmp/mtest-220875/.s.monetdb.31764
+QUERY = CREATE OR REPLACE FUNCTION Gvar() RETURNS string
+

MonetDB: scoping - Small bugfix, if a declared variable outside ...

[Pedro Ferreira]

Changeset: 0dcbea380d0f for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0dcbea380d0f
Modified Files:
sql/server/rel_psm.c
sql/server/sql_mvc.h
sql/test/scoping/Tests/scoping02.sql
sql/test/scoping/Tests/scoping02.stable.out
Branch: scoping
Log Message:

Small bugfix, if a declared variable outside the outermost scope of a function 
may have the same name as parameter, because the scope is different


diffs (72 lines):

diff --git a/sql/server/rel_psm.c b/sql/server/rel_psm.c
--- a/sql/server/rel_psm.c
+++ b/sql/server/rel_psm.c
@@ -195,7 +195,7 @@ rel_psm_declare(mvc *sql, dnode *n, bool
if (sname)
return sql_error(sql, 01, 
SQLSTATE(42000) "DECLARE: Variables declared inside functions and procedures 
don't have a schema");
/* find if there's a parameter with the same 
name */
-   if ((a = sql_bind_param(sql, tname)))
+   if (sql->frame == 1 && (a = sql_bind_param(sql, 
tname)))
return sql_error(sql, 01, 
SQLSTATE(42000) "DECLARE: Variable '%s' declared as a parameter", tname);
/* check if we overwrite a scope local variable 
declare x; declare x; */
if (frame_find_var(sql, tname))
diff --git a/sql/server/sql_mvc.h b/sql/server/sql_mvc.h
--- a/sql/server/sql_mvc.h
+++ b/sql/server/sql_mvc.h
@@ -90,7 +90,7 @@ typedef struct sql_rel_view { /* CTEs */
 } sql_rel_view;
 
 typedef struct sql_var { /* Declared variables and parameters */
-   char *sname; /* Declared variables have a schema */
+   char *sname; /* Global variables have a schema */
char *name;
atom var;
 } sql_var;
diff --git a/sql/test/scoping/Tests/scoping02.sql 
b/sql/test/scoping/Tests/scoping02.sql
--- a/sql/test/scoping/Tests/scoping02.sql
+++ b/sql/test/scoping/Tests/scoping02.sql
@@ -52,3 +52,18 @@ BEGIN
DECLARE TABLE mytable (a int);
RETURN SELECT a FROM sys.mytable; --error table sys.mytable doesn't 
exist
 END;
+
+CREATE OR REPLACE FUNCTION scoping02(i INT) RETURNS INT 
+BEGIN
+   IF i = 1 THEN
+   DECLARE i int; --valid, i is redeclared on another scope
+   RETURN i;
+   END IF;
+   RETURN i;
+END;
+
+SELECT scoping02(vals) FROM (VALUES (1), (2)) AS vals(vals);
+   -- NULL
+   -- 2
+
+DROP FUNCTION scoping02(INT);
diff --git a/sql/test/scoping/Tests/scoping02.stable.out 
b/sql/test/scoping/Tests/scoping02.stable.out
--- a/sql/test/scoping/Tests/scoping02.stable.out
+++ b/sql/test/scoping/Tests/scoping02.stable.out
@@ -54,6 +54,22 @@ stdout of test 'scoping02` in directory 
 [ 2]
 #DROP FUNCTION scoping01(INT);
 #DROP TABLE sys.mytable;
+#CREATE OR REPLACE FUNCTION scoping02(i INT) RETURNS INT 
+#BEGIN
+#  IF i = 1 THEN
+#  DECLARE i int; --valid, i is redeclared on another scope
+#  RETURN i;
+#  END IF;
+#  RETURN i;
+#END;
+#SELECT scoping02(vals) FROM (VALUES (1), (2)) AS vals(vals);
+% . # table_name
+% %4 # name
+% int # type
+% 1 # length
+[ NULL ]
+[ 2]
+#DROP FUNCTION scoping02(INT);
 
 # 11:27:58 >  
 # 11:27:58 >  "Done."
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Jun2020 - More future-proof initializations

[Pedro Ferreira]

Changeset: 3f09f346c6e2 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3f09f346c6e2
Modified Files:
sql/backends/monet5/mal_backend.c
sql/backends/monet5/sql_gencode.c
sql/backends/monet5/sql_statement.c
sql/common/sql_types.c
sql/server/rel_exp.c
sql/server/sql_atom.c
sql/server/sql_query.c
Branch: Jun2020
Log Message:

More future-proof initializations


diffs (223 lines):

diff --git a/sql/backends/monet5/mal_backend.c 
b/sql/backends/monet5/mal_backend.c
--- a/sql/backends/monet5/mal_backend.c
+++ b/sql/backends/monet5/mal_backend.c
@@ -13,17 +13,12 @@
 backend *
 backend_reset(backend *b)
 {
-   b->out = b->client->fdout;
-   b->language = 0;
-   b->depth = 0;
-
-   b->vtop = 0;
-   b->q = NULL;
-   b->mb = NULL;
-   b->mvc_var = 0;
-   b->cur_append = 0;
-   b->first_statement_generated = false;
-   b->output_format = OFMT_CSV;
+   *b = (backend) {
+   .mvc = b->mvc,
+   .client = b->client,
+   .out = b->client->fdout,
+   .output_format = OFMT_CSV,
+   };
return b;
 }
 
@@ -32,14 +27,12 @@ backend_create(mvc *m, Client c)
 {
backend *b = MNEW(backend);
 
-   if( b== NULL)
+   if (!b)
return NULL;
-   b->mvc = m;
-   b->client = c;
-   b->mvc_var = 0;
-   b->cur_append = 0;
-   b->first_statement_generated = false;
-   b->output_format = OFMT_CSV;
+   *b = (backend) {
+   .mvc = m,
+   .client = c,
+   };
return backend_reset(b);
 }
 
@@ -48,4 +41,3 @@ backend_destroy(backend *b)
 {
_DELETE(b);
 }
-
diff --git a/sql/backends/monet5/sql_gencode.c 
b/sql/backends/monet5/sql_gencode.c
--- a/sql/backends/monet5/sql_gencode.c
+++ b/sql/backends/monet5/sql_gencode.c
@@ -194,7 +194,7 @@ static int
 
for (n = rel_ops->h; n && !curBlk->errors; n = n->next) {
sql_exp *e = n->data;
-   sql_subtype *t = >tpe;
+   sql_subtype *t = exp_subtype(e);
int type = t->type->localtype;
int varid = 0;
char *buf;
diff --git a/sql/backends/monet5/sql_statement.c 
b/sql/backends/monet5/sql_statement.c
--- a/sql/backends/monet5/sql_statement.c
+++ b/sql/backends/monet5/sql_statement.c
@@ -215,22 +215,12 @@ static stmt *
 stmt_create(sql_allocator *sa, st_type type)
 {
stmt *s = SA_NEW(sa, stmt);
-   if(!s)
+
+   if (!s)
return NULL;
-
-   s->type = type;
-   s->op1 = NULL;
-   s->op2 = NULL;
-   s->op3 = NULL;
-   s->op4.lval = NULL;
-   s->flag = 0;
-   s->nrcols = 0;
-   s->key = 0;
-   s->aggr = 0;
-   s->nr = 0;
-   s->partition = 0;
-   s->tname = s->cname = NULL;
-   s->cand = NULL;
+   *s = (stmt) {
+   .type = type,
+   };
return s;
 }
 
diff --git a/sql/common/sql_types.c b/sql/common/sql_types.c
--- a/sql/common/sql_types.c
+++ b/sql/common/sql_types.c
@@ -130,17 +130,13 @@ bool is_commutative(const char *fnm)
 void
 base_init(sql_allocator *sa, sql_base * b, sqlid id, int flags, const char 
*name)
 {
-   b->id = id;
-
assert(sa);
-   b->wtime = 0;
-   b->rtime = 0;
-   b->stime = 0;
-   b->flags = flags;
-   b->name = NULL;
-   b->refcnt = 1;
-   if (name)
-   b->name = sa_strdup(sa,name);
+   *b = (sql_base) {
+   .id = id,
+   .flags = flags,
+   .refcnt = 1,
+   .name = (name) ? sa_strdup(sa, name) : NULL,
+   };
 }
 
 void
diff --git a/sql/server/rel_exp.c b/sql/server/rel_exp.c
--- a/sql/server/rel_exp.c
+++ b/sql/server/rel_exp.c
@@ -104,35 +104,15 @@ compare2range( int l, int r )
 }
 
 static sql_exp * 
-exp_create(sql_allocator *sa, int type ) 
+exp_create(sql_allocator *sa, int type) 
 {
sql_exp *e = SA_NEW(sa, sql_exp);
 
-   if (e == NULL)
+   if (!e)
return NULL;
-   e->type = (expression_type)type;
-   e->alias.label = 0;
-   e->alias.name = NULL;
-   e->alias.rname = NULL;
-   e->f = e->l = e->r = NULL;
-   e->flag = 0;
-   e->card = 0;
-   e->freevar = 0;
-   e->intern = 0;
-   e->anti = 0;
-   e->semantics = 0;
-   e->ascending = 0;
-   e->nulls_last = 0;
-   e->distinct = 0;
-   e->zero_if_empty = 0;
-   e->need_no_nil = 0;
-   e->has_no_nil = 0;
-   e->base = 0;
-   e->ref = 0;
-   e->used = 0;
-   e->tpe.type = NULL;
-   e->tpe.digits = e->tpe.scale = 0;
-   e->p = NULL;
+   *e = (sql_exp) {
+   .type = (expression_type) type,
+   };
return e;
 }
 
@@ -2406,7 +2386,7 @@ exp_flatten(mvc *sql, sql_exp *e)
} else if (e->type == e_convert) {
atom *v = exp_flatten(sql, e->l); 
 
- 

MonetDB: Jun2020 - Increased maximum name length for threads, se...

[Sjoerd Mullender]

Changeset: a33c7d85df44 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a33c7d85df44
Modified Files:
gdk/gdk.h
gdk/gdk_bat.c
gdk/gdk_bbp.c
gdk/gdk_hash.c
gdk/gdk_imprints.c
gdk/gdk_orderidx.c
gdk/gdk_storage.c
gdk/gdk_system.c
gdk/gdk_system.h
gdk/gdk_utils.c
monetdb5/mal/mal_client.c
monetdb5/mal/mal_dataflow.c
monetdb5/modules/mal/mal_mapi.c
monetdb5/modules/mal/tablet.c
Branch: Jun2020
Log Message:

Increased maximum name length for threads, semaphores and locks.
Done properly.


diffs (299 lines):

diff --git a/gdk/gdk.h b/gdk/gdk.h
--- a/gdk/gdk.h
+++ b/gdk/gdk.h
@@ -1711,7 +1711,7 @@ typedef struct threadStruct {
 * into this array + 1 (0 is
 * invalid) */
ATOMIC_TYPE pid;/* thread id, 0 = unallocated */
-   char name[16];
+   char name[MT_NAME_LEN];
void *data[THREADDATA];
uintptr_t sp;
 } *Thread;
diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c
--- a/gdk/gdk_bat.c
+++ b/gdk/gdk_bat.c
@@ -121,7 +121,7 @@ BATcreatedesc(oid hseq, int tt, bool hea
bn->tvheap->parentid = bn->batCacheid;
bn->tvheap->farmid = BBPselectfarm(role, bn->ttype, varheap);
}
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "BATlock%d", bn->batCacheid); /* fits */
MT_lock_init(>batIdxLock, name);
bn->batDirtydesc = true;
diff --git a/gdk/gdk_bbp.c b/gdk/gdk_bbp.c
--- a/gdk/gdk_bbp.c
+++ b/gdk/gdk_bbp.c
@@ -1125,7 +1125,7 @@ BBPreadEntries(FILE *fp, unsigned bbpver
bn->batCount = (BUN) count;
bn->batInserted = bn->batCount;
bn->batCapacity = (BUN) capacity;
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "BATlock%d", bn->batCacheid); /* 
fits */
MT_lock_init(>batIdxLock, name);
 
diff --git a/gdk/gdk_hash.c b/gdk/gdk_hash.c
--- a/gdk/gdk_hash.c
+++ b/gdk/gdk_hash.c
@@ -957,7 +957,7 @@ BAThash(BAT *b)
if (BBP_status(b->batCacheid) & BBPEXISTING && !b->theap.dirty 
&& !GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "hashsync%d", 
b->batCacheid);
MT_lock_unset(>batIdxLock);
if (MT_create_thread(, BAThashsync, b,
diff --git a/gdk/gdk_imprints.c b/gdk/gdk_imprints.c
--- a/gdk/gdk_imprints.c
+++ b/gdk/gdk_imprints.c
@@ -534,7 +534,7 @@ BATimprints(BAT *b)
!GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "impssync%d", 
b->batCacheid);
if (MT_create_thread(, BATimpsync, b,
 MT_THR_DETACHED, name) < 0)
diff --git a/gdk/gdk_orderidx.c b/gdk/gdk_orderidx.c
--- a/gdk/gdk_orderidx.c
+++ b/gdk/gdk_orderidx.c
@@ -169,7 +169,7 @@ persistOIDX(BAT *b)
!GDKinmemory()) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "oidxsync%d", b->batCacheid);
if (MT_create_thread(, BATidxsync, b,
 MT_THR_DETACHED, name) < 0)
@@ -479,7 +479,7 @@ GDKmergeidx(BAT *b, BAT**a, int n_ar)
b->batInserted == b->batCount) {
MT_Id tid;
BBPfix(b->batCacheid);
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "oidxsync%d", b->batCacheid);
if (MT_create_thread(, BATidxsync, b,
 MT_THR_DETACHED, name) < 0)
diff --git a/gdk/gdk_storage.c b/gdk/gdk_storage.c
--- a/gdk/gdk_storage.c
+++ b/gdk/gdk_storage.c
@@ -705,7 +705,7 @@ BATmsync(BAT *b)
arg->h = >theap;
BBPfix(b->batCacheid);
 #ifdef MSYNC_BACKGROUND
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, sizeof(name), "msync%d", b->batCacheid);
if (MT_create_thread(, BATmsyncImplementation, arg,
 MT_THR_DETACHED, name) < 0) {
@@ -724,7 +724,7 @@ BATmsync(BAT *b)
arg->h = b->tvheap;
BBPfix(b->batCacheid);
 #ifdef MSYNC_BACKGROUND
-   char name[16];
+   char name[MT_NAME_LEN];
snprintf(name, 

MonetDB: Jun2020 - merged

[Niels Nes]

Changeset: e3258a13b708 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e3258a13b708
Modified Files:
sql/test/Tests/hot-snapshot.py
Branch: Jun2020
Log Message:

merged


diffs (38 lines):

diff --git a/sql/test/Tests/hot-snapshot.py b/sql/test/Tests/hot-snapshot.py
--- a/sql/test/Tests/hot-snapshot.py
+++ b/sql/test/Tests/hot-snapshot.py
@@ -18,7 +18,6 @@ except ImportError:
 import os
 import shutil
 import socket
-import time
 import tarfile
 
 import pymonetdb
@@ -50,8 +49,6 @@ def main():
 
 # start the server
 with process.server(dbname=mydb, mapiport = mapi_port, 
stdin=process.PIPE) as server:
-time.sleep(2)
-
 # connection 1 creates, inserts, commits and inserts uncommitted
 conn1 = pymonetdb.connect(
 database=server.dbname, hostname='localhost',
@@ -84,7 +81,8 @@ def main():
 cur2.close()
 conn2.close()
 server.terminate()
-time.sleep(1)
+server.wait()
+
 shutil.rmtree(mydbdir)
 
 # and extract the tarname
@@ -111,6 +109,7 @@ def main():
 cur3.close()
 conn3.close()
 server.terminate()
+server.wait()
 
 finally:
 if os.path.exists(mydbdir):
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Jun2020 - some semaphore names grow too large.

[Niels Nes]

Changeset: 36b912b0477b for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=36b912b0477b
Modified Files:
gdk/gdk_utils.c
Branch: Jun2020
Log Message:

some semaphore names grow too large.


diffs (12 lines):

diff --git a/gdk/gdk_utils.c b/gdk/gdk_utils.c
--- a/gdk/gdk_utils.c
+++ b/gdk/gdk_utils.c
@@ -1399,7 +1399,7 @@ THRcreate(void (*f) (void *), void *arg,
Thread s;
struct THRstart *t;
static ATOMIC_TYPE ctr = ATOMIC_VAR_INIT(0);
-   char semname[16];
+   char semname[32];
int len;
 
if ((t = GDKmalloc(sizeof(*t))) == NULL)
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: unlock - fix windows/intel compilation issues

[Niels Nes]

Changeset: 699eb5507d9e for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=699eb5507d9e
Modified Files:
gdk/gdk_logger.c
sql/storage/bat/bat_storage.c
Branch: unlock
Log Message:

fix windows/intel compilation issues


diffs (90 lines):

diff --git a/gdk/gdk_logger.c b/gdk/gdk_logger.c
--- a/gdk/gdk_logger.c
+++ b/gdk/gdk_logger.c
@@ -456,7 +456,7 @@ la_bat_updates(logger *lg, logaction *la
BATiter vi = bat_iterator(la->b);
BUN p, q;

-   for (p=0, q = la->offset; p<(BUN)la->nr; p++, 
q++) {
+   for (p=0, q = (BUN)la->offset; p<(BUN)la->nr; 
p++, q++) {
const void *t = BUNtail(vi, p);

if (q < cnt) {
@@ -473,7 +473,7 @@ la_bat_updates(logger *lg, logaction *la
}
}
}
-   cnt = la->offset + la->nr;
+   cnt = (BUN)(la->offset + la->nr);
if (la_bat_update_count(lg, la->cid, cnt) != GDK_SUCCEED) 
return GDK_FAIL;
} else if (!lg->flushing && la->type == LOG_UPDATE) {
@@ -1226,9 +1226,9 @@ bm_subcommit(logger *lg)
n[i++] = col;
}
/* now commit catalog, so it's also up to date on disk */
-   sizes[i] = lg->cnt;
+   sizes[i] = (BUN)lg->cnt;
n[i++] = catalog_bid->batCacheid;
-   sizes[i] = lg->cnt;
+   sizes[i] = (BUN)lg->cnt;
n[i++] = catalog_id->batCacheid;
sizes[i] = BATcount(dcatalog); /* todo ! */
n[i++] = dcatalog->batCacheid;
diff --git a/sql/storage/bat/bat_storage.c b/sql/storage/bat/bat_storage.c
--- a/sql/storage/bat/bat_storage.c
+++ b/sql/storage/bat/bat_storage.c
@@ -1258,7 +1258,7 @@ load_bat(sql_delta *bat, int type, sqlid
 }
 
 static int
-log_create_delta(sql_delta *bat, oid id) 
+log_create_delta(sql_delta *bat, sqlid id) 
 {
int res = LOG_OK;
gdk_return ok;
@@ -1597,7 +1597,7 @@ create_del(sql_trans *tr, sql_table *t)
 }
 
 static int
-log_create_dbat( sql_dbat *bat, oid id)
+log_create_dbat( sql_dbat *bat, sqlid id)
 {
BAT *b;
gdk_return ok;
@@ -1627,7 +1627,7 @@ log_create_del(sql_trans *tr, sql_table 
 }
 
 static int
-log_destroy_delta(sql_trans *tr, sql_delta *b, oid id)
+log_destroy_delta(sql_trans *tr, sql_delta *b, sqlid id)
 {
gdk_return ok = GDK_SUCCEED;
 
@@ -1815,7 +1815,7 @@ destroy_del(sql_trans *tr, sql_table *t)
 }
 
 static int 
-log_destroy_dbat(sql_trans *tr, sql_dbat *bat, oid id)
+log_destroy_dbat(sql_trans *tr, sql_dbat *bat, sqlid id)
 {
gdk_return ok = GDK_SUCCEED;
 
@@ -2508,7 +2508,7 @@ update_table(sql_trans *tr, sql_table *f
 }
 
 static int 
-tr_log_delta( sql_trans *tr, sql_delta *cbat, int cleared, oid id)
+tr_log_delta( sql_trans *tr, sql_delta *cbat, int cleared, sqlid id)
 {
gdk_return ok = GDK_SUCCEED;
BAT *ins;
@@ -2550,7 +2550,7 @@ tr_log_delta( sql_trans *tr, sql_delta *
 }
 
 static int
-tr_log_dbat(sql_trans *tr, sql_dbat *fdb, int cleared, oid id)
+tr_log_dbat(sql_trans *tr, sql_dbat *fdb, int cleared, sqlid id)
 {
gdk_return ok = GDK_SUCCEED;
BAT *db = NULL;
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Jun2020 - Don't sleep, wait for server to terminate

[Pedro Ferreira]

Changeset: 594f3f21f7ee for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=594f3f21f7ee
Modified Files:
sql/test/Tests/hot-snapshot.py
Branch: Jun2020
Log Message:

Don't sleep, wait for server to terminate


diffs (38 lines):

diff --git a/sql/test/Tests/hot-snapshot.py b/sql/test/Tests/hot-snapshot.py
--- a/sql/test/Tests/hot-snapshot.py
+++ b/sql/test/Tests/hot-snapshot.py
@@ -18,7 +18,6 @@ except ImportError:
 import os
 import shutil
 import socket
-import time
 import tarfile
 
 import pymonetdb
@@ -50,8 +49,6 @@ def main():
 
 # start the server
 with process.server(dbname=mydb, mapiport = mapi_port, 
stdin=process.PIPE) as server:
-time.sleep(2)
-
 # connection 1 creates, inserts, commits and inserts uncommitted
 conn1 = pymonetdb.connect(
 database=server.dbname, hostname='localhost',
@@ -84,7 +81,8 @@ def main():
 cur2.close()
 conn2.close()
 server.terminate()
-time.sleep(1)
+server.wait()
+
 shutil.rmtree(mydbdir)
 
 # and extract the tarname
@@ -111,6 +109,7 @@ def main():
 cur3.close()
 conn3.close()
 server.terminate()
+server.wait()
 
 finally:
 if os.path.exists(mydbdir):
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: unlock - fix compilation issues

[Niels Nes]

Changeset: aca37f83013a for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=aca37f83013a
Modified Files:
gdk/gdk_logger.c
sql/storage/store.c
Branch: unlock
Log Message:

fix compilation issues


diffs (95 lines):

diff --git a/gdk/gdk_logger.c b/gdk/gdk_logger.c
--- a/gdk/gdk_logger.c
+++ b/gdk/gdk_logger.c
@@ -434,7 +434,7 @@ la_bat_updates(logger *lg, logaction *la
 
if (!lg->flushing) {
/* handle offset 0 ie clear */
-   if (0 && la->offset == 0 && cnt)
+   if (/* DISABLES CODE */ (0) && la->offset == 0 && cnt)
BATclear(b, true);
/* handle offset */
if (cnt <= (BUN)la->offset) {
@@ -602,7 +602,7 @@ logger_write_new_types(logger *lg, FILE 
id++;
}
/* second the var sized types */
-   id=129; // start after nil 
+   id=-127; // start after nil 
for (int i=0;isaved_id);
+   return logger_cleanup(lg, lg->saved_id);
}
return res == LOG_ERR ? GDK_FAIL : GDK_SUCCEED;
 }
diff --git a/sql/storage/store.c b/sql/storage/store.c
--- a/sql/storage/store.c
+++ b/sql/storage/store.c
@@ -2071,10 +2071,10 @@ store_init(int debug, store_type store, 
return store_load(stk);
 }
 
+#if 0
 static int
 store_needs_vacuum( sql_trans *tr )
 {
-   return 0;
size_t max_dels = GDKdebug & FORCEMITOMASK ? 1 : 128;
sql_schema *s = find_sql_schema(tr, "sys");
node *n;
@@ -2116,6 +2116,7 @@ store_vacuum( sql_trans *tr )
}
return 0;
 }
+#endif
 
 // All this must only be accessed while holding the bs_lock.
 // The exception is flush_now, which can be set by anyone at any
@@ -2269,7 +2270,7 @@ store_apply_deltas(void)
trans_cleanup(gtrans);
 
res = logger_funcs.flush();
-   if (0 && /*gtrans->sa->nr > 40 &&*/ !(ATOMIC_GET(_sessions)) /* only 
save when there are no dependencies on the gtrans */) { /* TODO need better 
estimate */
+   if (/* DISABLES CODE */ (0) && /*gtrans->sa->nr > 40 &&*/ 
!(ATOMIC_GET(_sessions)) /* only save when there are no dependencies on the 
gtrans */) { /* TODO need better estimate */
sql_trans *ntrans = sql_trans_create(gtrans->stk, gtrans, NULL, 
false);
 
trans_init(ntrans, ntrans->stk, gtrans);
@@ -2358,7 +2359,6 @@ idle_manager(void)
 
MT_thread_setworking("sleeping");
while (!GDKexiting()) {
-   sql_session *s;
int t;
 
for (t = timeout; t > 0; t -= sleeptime) {
@@ -2369,12 +2369,13 @@ idle_manager(void)
/* cleanup any collected intermediate storage */
store_funcs.cleanup();
MT_lock_set(_lock);
-   if (ATOMIC_GET(_nr_active) || GDKexiting() || 
!store_needs_vacuum(gtrans)) {
+   if (ATOMIC_GET(_nr_active) || GDKexiting() /*|| 
!store_needs_vacuum(gtrans)*/) {
MT_lock_unset(_lock);
continue;
}
 
-   s = sql_session_create(gtrans->stk, 0);
+#if 0
+   sql_session *s = sql_session_create(gtrans->stk, 0);
if (!s) {
MT_lock_unset(_lock);
continue;
@@ -2385,7 +2386,7 @@ idle_manager(void)
sql_trans_commit(s->tr);
sql_trans_end(s, 1);
sql_session_destroy(s);
-
+#endif
MT_lock_unset(_lock);
MT_thread_setworking("sleeping");
}
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list