date:20170105

MonetDB: wlcr - mereg with default

2017-01-05 Thread Martin Kersten

Changeset: c74f22dee6da for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=c74f22dee6da
Added Files:
README.rst
Removed Files:
README
Modified Files:
HowToStart.rst
Makefile.ag
clients/Tests/MAL-signatures.stable.out
clients/Tests/MAL-signatures.stable.out.int128
clients/Tests/exports.stable.out
monetdb5/ChangeLog
monetdb5/extras/rapi/Tests/rapi08.malC
monetdb5/mal/mal_authorize.c
monetdb5/mal/mal_authorize.h
monetdb5/mal/mal_module.c
monetdb5/mal/mal_session.c
monetdb5/modules/atoms/batxml.c
monetdb5/modules/kernel/batcolor.c
monetdb5/modules/kernel/batstr.c
monetdb5/modules/mal/clients.c
sql/backends/monet5/sql_user.c
sql/backends/monet5/vaults/bam/bam_wrapper.c
sql/server/sql_parser.y
tools/merovingian/daemon/controlrunner.c
tools/mserver/mserver5.c
tools/mserver/shutdowntest.c
Branch: wlcr
Log Message:

mereg with default


diffs (truncated from 1084 to 300 lines):

diff --git a/HowToStart.rst b/HowToStart.rst
--- a/HowToStart.rst
+++ b/HowToStart.rst
@@ -24,7 +24,7 @@ This document assumes that you are plann
 installing MonetDB on a Unix-like system (e.g., Linux, IRIX, Solaris,
 AIX, Mac OS X/Darwin, or CYGWIN).  For compilation and installation on
 a native Windows system (NT, 2000, XP) see the instructions in the
-file `../buildtools/doc/windowsbuild.rst`__.
+file `buildtools/doc/windowsbuild.rst`__.
 
 __ http://dev.monetdb.org/downloads/
 __ Windows-Installation.html
diff --git a/Makefile.ag b/Makefile.ag
--- a/Makefile.ag
+++ b/Makefile.ag
@@ -13,7 +13,7 @@ SUBDIRS = buildtools common clients \
HAVE_TESTING?testing \
 
 EXTRA_DIST = bootstrap configure configure.ac configure.ag libversions \
-MonetDB.spec rpm.mk.in COPYING README license.txt 
HowToStart.rst
+MonetDB.spec rpm.mk.in COPYING README.rst license.txt 
HowToStart.rst
 
 EXTRA_DIST_DIR = NT debian MacOSX
 
diff --git a/README b/README
deleted file mode 100644
--- a/README
+++ /dev/null
@@ -1,26 +0,0 @@
-The MonetDB Database System
-===
-
-The MonetDB database is developed by the CWI database research group
-(see http://www.monetdb.org/).
-
-Via the MonetDB project we have brought the MonetDB system in open source,
-where it is accessible at http://www.monetdb.org/Downloads/
-
-The MonetDB database system is a high-performance database kernel for
-query-intensive applications. The MonetDB kernel works together with an
-SQL frontend that is in a separate CVS module.
-
-If you got a source distribution, please compile and install MonetDB first,
-following the instructions in the file 'HowToStart.rst' (for Unix)
-or 'buildtools/doc/windowsbuild.rst' (for Windows).
-
-
-Copyright Notice
-
-
-This Source Code Form is subject to the terms of the Mozilla Public
-License, v. 2.0.  If a copy of the MPL was not distributed with this
-file, You can obtain one at http://mozilla.org/MPL/2.0/.
-
-Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
diff --git a/README.rst b/README.rst
new file mode 100644
--- /dev/null
+++ b/README.rst
@@ -0,0 +1,34 @@
+The MonetDB Database System
+===
+
+The MonetDB database is developed by the CWI database research group
+(see http://www.monetdb.org/).
+
+Via the MonetDB project we have brought the MonetDB system in open source,
+where it is accessible at http://www.monetdb.org/Downloads/
+
+The MonetDB database system is a high-performance database kernel for
+query-intensive applications. The MonetDB source can be found at our `Mercurial
+server`__. There is also a `github mirror`__ that is updated once a day.
+
+.. _MonetDB: http://dev.monetdb.org/hg/MonetDB/
+__ MonetDB_
+
+.. _github: https://github.com/MonetDB/MonetDB
+__ github_
+
+If you got a source distribution, please compile and install MonetDB first,
+following the instructions in the file `HowToStart.rst`__ (for Unix)
+or `buildtools/doc/windowsbuild.rst`__ (for Windows).
+
+__ HowToStart.rst
+__ buildtools/doc/windowsbuild.rst
+
+Copyright Notice
+
+
+This Source Code Form is subject to the terms of the Mozilla Public
+License, v. 2.0.  If a copy of the MPL was not distributed with this
+file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
diff --git a/clients/Tests/MAL-signatures.stable.out 
b/clients/Tests/MAL-signatures.stable.out
--- a/clients/Tests/MAL-signatures.stable.out
+++ b/clients/Tests/MAL-signatures.stable.out
@@ -8654,8 +8654,6 @@ Ready.
 [ "url",   "url",  "command url.url(s:url):url ",  "URLnoop;", "Create 
an URL from a string literal"   ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main",

MonetDB: wlcr - Create the replica server process

2017-01-05 Thread Martin Kersten

Changeset: 2e0ad40d1449 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=2e0ad40d1449
Modified Files:
sql/backends/monet5/Makefile.ag
sql/backends/monet5/sql_wlcr.c
sql/backends/monet5/sql_wlcr.h
sql/backends/monet5/sql_wlcr.mal
sql/scripts/60_wlcr.sql
Branch: wlcr
Log Message:

Create the replica server process
Prepare for processing the wlcr log files at the replica.


diffs (284 lines):

diff --git a/sql/backends/monet5/Makefile.ag b/sql/backends/monet5/Makefile.ag
--- a/sql/backends/monet5/Makefile.ag
+++ b/sql/backends/monet5/Makefile.ag
@@ -64,7 +64,7 @@ headers_mal = {
HEADERS = mal
DIR = libdir/monetdb5
SOURCES = sql_aggr_bte.mal  sql_aggr_flt.mal sql_aggr_dbl.mal  
sql_aggr_int.mal  sql_aggr_lng.mal \
-   sql_aggr_sht.mal  sql_decimal.mal  sql_inspect.mal sql_wlcr.mal\
+   sql_aggr_sht.mal  sql_decimal.mal  sql_inspect.mal sql_wlcr.mal 
\
sql_rank.mal sqlcatalog.mal sql_transaction.mal  sql.mal
 }
 
diff --git a/sql/backends/monet5/sql_wlcr.c b/sql/backends/monet5/sql_wlcr.c
--- a/sql/backends/monet5/sql_wlcr.c
+++ b/sql/backends/monet5/sql_wlcr.c
@@ -16,91 +16,153 @@
  * Alternatively you start with an empty database.
  *
  * Since the wlcr files can be stored anywhere, the full path should be given.
+ *
+ * At any time there can only be on choice to interpret the files.
  */
 #include "monetdb_config.h"
 #include "sql.h"
 #include "wlcr.h"
 #include "sql_wlcr.h"
+#include "mal_client.h"
+
+#define WLCR_REPLAY 1
+#define WLCR_SYNC 2
+static int wlcr_mode;
 
 static str wlcr_master;
 static int wlcr_replaythreshold;
-static int wlcr_replaybatch;
+static int wlcr_replaybatches;
+
+static MT_Id wlcr_thread;
 
 static str
-WLCRinit(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
+WLCRreplayinit(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
 {
-int i = 1, j,k,l;
+int i = 1, j,k;
char path[PATHLENGTH];
+   str dbname,dir;
FILE *fd;
 
(void) cntxt;
(void) k;
-   (void) l;
 
-wlcr_master = GDKgetenv("gdk_master");
-if (i< pci->argc+1 && getArgType(mb, pci, i) == TYPE_str){
-wlcr_master =  *getArgReference_str(stk,pci,i);
+if (getArgType(mb, pci, i) == TYPE_str){
+dbname =  *getArgReference_str(stk,pci,i);
 i++;
 }
-   if( wlcr_master == NULL){
-   throw(SQL,"wlcr.init","Can not access the wlcr directory");
+   if( dbname == NULL){
+   throw(SQL,"wlcr.init","Master database name missing.");
}
-   snprintf(path,PATHLENGTH,"%s%cwlcr", wlcr_master, DIR_SEP);
-   mnstr_printf(cntxt->fdout,"#Testing '%s'\n", path);
+   snprintf(path,PATHLENGTH,"..%c%s",DIR_SEP,dbname);
+   dir = GDKfilepath(0,path,"master",0);
+   wlcr_master = GDKstrdup(dir);
+   mnstr_printf(cntxt->fdout,"#WLCR master '%s'\n", wlcr_master);
+   snprintf(path,PATHLENGTH,"%s%cwlcr", dir, DIR_SEP);
+   mnstr_printf(cntxt->fdout,"#Testing access to master '%s'\n", path);
fd = fopen(path,"r");
if( fd == NULL){
-   throw(SQL,"wlcr.init","Can not access '%s'\n",path);
+   throw(SQL,"wlcr.init","Can not access master control file 
'%s'\n",path);
}
-   if( fscanf(fd,"%d %d %d", ,,) != 3){
+   if( fscanf(fd,"%d %d", ,) != 2){
throw(SQL,"wlcr.init","'%s' does not have proper number of 
arguments\n",path);
}
-   wlcr_replaybatch = j;
+   wlcr_replaybatches = j;
 
-if ( i < pci->argc+1 && getArgType(mb, pci, i) == TYPE_int){
+if ( i < pci->argc && getArgType(mb, pci, i) == TYPE_int){
 wlcr_replaythreshold = *getArgReference_int(stk,pci,i);
}
return MAL_SUCCEED;
 }
 
+void
+WLCRprocess(void *arg)
+{  
+   Client cntxt = (Client) arg;
+   int i;
+   char path[PATHLENGTH];
+   stream *fd;
+   Client c;
+
+   c =MCforkClient(cntxt);
+   if( c == 0){
+   GDKerror("Could not create user for WLCR process\n");
+   return;
+   }
+c->prompt = GDKstrdup("");  /* do not produce visible prompts */
+c->promptlength = 0;
+c->listing = 0;
+
+   mnstr_printf(cntxt->fdout,"#Ready to start the replayagainst '%s' 
batches %d threshold %d", wlcr_master, wlcr_replaybatches, 
wlcr_replaythreshold);
+   for( i= 0; i < wlcr_replaybatches; i++){
+   snprintf(path,PATHLENGTH,"%s%cwlcr_%06d", wlcr_master, 
DIR_SEP,i);
+   mnstr_printf(cntxt->fdout,"#WLCR processing %s\n",path);
+   fd= open_rstream(path);
+   if( c->fdin == NULL || MCpushClientInput(c, bstream_create(fd, 
128 * BLOCK), 0, "") < 0){
+   mnstr_printf(cntxt->fdout,"#wlcr.replay:'%s' can not be 
accessed \n",path);
+   }
+   c->yycur = 0;
+   // preload the complete file
+   // now parse the file line by line

MonetDB: wlcr - fix compilation

2017-01-05 Thread Martin Kersten

Changeset: 48a424fe4f7c for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=48a424fe4f7c
Modified Files:
monetdb5/optimizer/opt_wlcr.c
monetdb5/optimizer/opt_wlcr.h
Branch: wlcr
Log Message:

fix compilation


diffs (36 lines):

diff --git a/monetdb5/optimizer/opt_wlcr.c b/monetdb5/optimizer/opt_wlcr.c
--- a/monetdb5/optimizer/opt_wlcr.c
+++ b/monetdb5/optimizer/opt_wlcr.c
@@ -12,7 +12,6 @@
  * We don't need the actual name of the objects
  */
 #include "monetdb_config.h"
-#include "wlcr.h"
 #include "opt_wlcr.h"
 
 
diff --git a/monetdb5/optimizer/opt_wlcr.h b/monetdb5/optimizer/opt_wlcr.h
--- a/monetdb5/optimizer/opt_wlcr.h
+++ b/monetdb5/optimizer/opt_wlcr.h
@@ -6,16 +6,16 @@
  * Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
  */
 
-#ifndef _WLCR_H
-#define _WLCR_H
-/* #define _WLCR_DEBUG_*/
+#ifndef _OPT_WLCR_H
+#define _OPT_WLCR_H
+/* #define _OPT_WLCR_DEBUG_*/
 
 #include "mal_interpreter.h"
 #include "mal_scenario.h"
-#include "opt_wlcr.h"
+#include "wlcr.h"
 #include "opt_support.h"
 #include "opt_prelude.h"
 
 mal_export int OPTwlcrImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr 
stk, InstrPtr pci);
 
-#endif /* _WLCR_H */
+#endif /* _OPT_WLCR_H */
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: wlcr - Enable startup testing

2017-01-05 Thread Martin Kersten

Changeset: 9ba04037017c for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9ba04037017c
Modified Files:
monetdb5/modules/mal/wlcr.c
monetdb5/modules/mal/wlcr.h
monetdb5/modules/mal/wlcr.mal
Branch: wlcr
Log Message:

Enable startup testing
All logs are stored in dbname/master/wlcr_*
The collection is extended during each server restart.


diffs (truncated from 436 to 300 lines):

diff --git a/monetdb5/modules/mal/wlcr.c b/monetdb5/modules/mal/wlcr.c
--- a/monetdb5/modules/mal/wlcr.c
+++ b/monetdb5/modules/mal/wlcr.c
@@ -11,46 +11,46 @@
  * This module collects the workload-capture-replay statements during 
transaction execution. 
  * It is used primarilly for replication management and workload replay
  *
- * The goal is to maintain a replica of a master database. All data of the 
master
+ * The goal is to create and use a replica of a master database. All data of 
the master
  * is basically only available for read only access. Accidental corruption of 
this
  * data is avoided by setting ownership and access properties at the SQL level 
in the replica.
  *
  *
  * IMPLEMENTATION
  *
- * The replica directory should be on a shared (global) file system.
- * As default we use dbfarm/master.
+ * A database can be set into 'master' mode only once.
+ * As default we use dbfarm/master to collect the necessary information.
  *
  * The binary dump for the database snapshot should be stored there in  
master/bat.
  * The associated log files are stored as master/wlcr.
  * Creation and restore of a snapshot should be a  monetdb option. TODO
  *
  * Replication management start when you run the command 
- * CALL wlcr.master("(full)path to snapshot dir")
- * It can also be passed as a command line parameter 
- * --set wlcr_dir="(full)path to snapshot dir"
+ * CALL wlcr.master()
+ * It can also be activated as a command line parameter 
+ * --set wlcr=yes
  *
  * Each wlcr log file contains a serial log for a transaction batch. 
- * Each job is identified by the owner of the query, the snapshot tag,
+ * Each job is identified by the owner of the query, 
  * commit/rollback, its starting time and runtime (in ms).
  *
  * Logging of queries can be further limited to those that satisfy a threshold.
- * CALL wlcr.master("(full)path to snapshot dir", threshold)
+ * CALL wlcr.master(threshold)
  * The threshold is given in milliseconds. A negative threshold leads to 
ignoring all queries.
  *
  * A replica server should issue the matching call
- * CALL wlcr.synchronize("(full)path to snapshot dir")
+ * CALL wlcr.synchronize("dbname")
  *
- * During synchronization only updates are executed for the user responsible 
for the call. 
+ * During synchronization only updates are executed.
  * Queries are simply ignored unless needed as replacement for update actions.
  *
  * The alternative is to replay the log
- * CALL wlcr.replay("(full)path to snapshot dir")
+ * CALL wlcr.replay("dbname")
  * In this mode all queries are executed under the credentials of the query 
owner, including those that lead to updates.
  *
  * Any failure encountered terminates the synchronization process, leaving a 
message in the merovingian log.
  *
- * The replay progress can be inspected using the function wlcr.drift() and 
wlcr.synced().
+ * The replay progress can be inspected using the function wlcr.synced().
  * The latter is true if all accessible log files have been processed.
  * 
  * The wlcr files purposely have a textual format derived from the MAL 
statements.
@@ -59,6 +59,8 @@
  * The integrity of the wlcr directories is critical. For now we assume that 
all batches are available. 
  * We should detect that wlcr.master() is issued after updates have taken 
place on the snapshot TODO.
  *
+ * The WLCR logs are always private a given thread
+ *
  */
 #include "monetdb_config.h"
 #include 
@@ -69,13 +71,13 @@ static MT_Lock wlcr_lock MT_LOCK_INI
 
 
 int wlcr_threshold = 0; // threshold (milliseconds) for keeping readonly 
queries
-str wlcr_snapshot= 0;  // name assigned to the snapshot
-int wlcr_batch = 0;// last job executed
+int wlcr_batch = 0;// last batch jon identifier 
+int wlcr_tid = 0;  // last transaction id
 
 static char *wlcr_name[]= {"","query","update","catalog"};
 
 static stream *wlcr_fd = 0;
-static str wlcr_dir = 0;
+str wlcr_dir = 0;
 
 /* The database snapshots are binary copies of the dbfarm/database/bat
  * New snapshots are created currently using the 'monetdb snapshot ' 
command
@@ -84,6 +86,81 @@ static str wlcr_dir = 0;
  *
  * The wlcr logs are stored in the snapshot directory as a time-stamped list
  */
+
+// creation of file and updating the version file should be atomic TODO!!!
+static str
+WLCRloggerfile(Client cntxt)
+{
+   char path[PATHLENGTH];
+   FILE *fd;
+
+   (void) cntxt;
+   snprintf(path,PATHLENGTH,"%s%cwlcr",wlcr_dir, DIR_SEP);
+   mnstr_printf(cntxt->fdout,"#WLCRloggerfile %s\n",wlcr_dir);
+   fd =

MonetDB: wlcr - Only call wlcr optimizer when needed.

2017-01-05 Thread Martin Kersten

Changeset: 9807fbeb0e97 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9807fbeb0e97
Modified Files:
monetdb5/optimizer/opt_wlcr.c
Branch: wlcr
Log Message:

Only call wlcr optimizer when needed.


diffs (37 lines):

diff --git a/monetdb5/optimizer/opt_wlcr.c b/monetdb5/optimizer/opt_wlcr.c
--- a/monetdb5/optimizer/opt_wlcr.c
+++ b/monetdb5/optimizer/opt_wlcr.c
@@ -12,6 +12,7 @@
  * We don't need the actual name of the objects
  */
 #include "monetdb_config.h"
+#include "wlcr.h"
 #include "opt_wlcr.h"
 
 
@@ -27,6 +28,8 @@ OPTwlcrImplementation(Client cntxt, MalB
(void) cntxt;
(void) stk; /* to fool compilers */
 
+   if( wlcr_dir == NULL)
+   goto wrapup;
old= mb->stmt;
limit= mb->stop;
slimit = mb->ssize;
@@ -59,7 +62,7 @@ OPTwlcrImplementation(Client cntxt, MalB
q= copyInstruction(p);
setModuleId(q, wlcrRef);
getArg(q,0) = newTmpVariable(mb,TYPE_any);
-   delArgument(q, 2);
+   delArgument(q, 3);
pushInstruction(mb,q);
} else
if( getModuleId(p) == sqlcatalogRef){
@@ -102,6 +105,7 @@ OPTwlcrImplementation(Client cntxt, MalB
printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL);
 #endif
 
+wrapup:
 snprintf(buf,256,"%-20s actions=%2d time=" LLFMT " 
usec","wlcr",updates,GDKusec() - usec);
 newComment(mb,buf);
return 1;
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: wlcr - Prepare for reading single MAL statements from s...

2017-01-05 Thread Martin Kersten

Changeset: 475960715bb1 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=475960715bb1
Modified Files:
monetdb5/mal/mal_import.c
monetdb5/mal/mal_parser.c
monetdb5/mal/mal_parser.h
monetdb5/mal/mal_session.c
Branch: wlcr
Log Message:

Prepare for reading single MAL statements from source


diffs (66 lines):

diff --git a/monetdb5/mal/mal_import.c b/monetdb5/mal/mal_import.c
--- a/monetdb5/mal/mal_import.c
+++ b/monetdb5/mal/mal_import.c
@@ -202,7 +202,7 @@ malInclude(Client c, str name, int listi
c->bak = NULL;
c->fdin = bstream_create(mal_init_stream, mal_init_len);
bstream_next(c->fdin);
-   parseMAL(c, c->curprg, 1);
+   parseMAL(c, c->curprg, 1, INT_MAX);
free(mal_init_buf);
free(mal_init_stream);
free(c->fdin);
@@ -219,7 +219,7 @@ malInclude(Client c, str name, int listi
c->yycur = 0;
c->bak = NULL;
if ((s = malLoadScript(c, filename, >fdin)) == 
MAL_SUCCEED) {
-   parseMAL(c, c->curprg, 1);
+   parseMAL(c, c->curprg, 1, INT_MAX);
bstream_destroy(c->fdin);
} else {
GDKfree(s); // not interested in error here
diff --git a/monetdb5/mal/mal_parser.c b/monetdb5/mal/mal_parser.c
--- a/monetdb5/mal/mal_parser.c
+++ b/monetdb5/mal/mal_parser.c
@@ -1744,7 +1744,7 @@ part3:
 #define BRKONERR if (curPrg->def->errors >= MAXERRORS) \
return curPrg->def->errors;
 int
-parseMAL(Client cntxt, Symbol curPrg, int skipcomments)
+parseMAL(Client cntxt, Symbol curPrg, int skipcomments, int lines)
 {
int cntrl = 0;
/*Symbol curPrg= cntxt->curprg;*/
@@ -1753,7 +1753,7 @@ parseMAL(Client cntxt, Symbol curPrg, in
 
echoInput(cntxt);
/* here the work takes place */
-   while ((c = currChar(cntxt))) {
+   while ((c = currChar(cntxt)) && lines-- > 0) {
switch (c) {
case '\n': case '\r': case '\f':
nextChar(cntxt);
diff --git a/monetdb5/mal/mal_parser.h b/monetdb5/mal/mal_parser.h
--- a/monetdb5/mal/mal_parser.h
+++ b/monetdb5/mal/mal_parser.h
@@ -20,7 +20,7 @@
 #define prevChar(X) if(X->yycur) X->yycur--
 
 mal_export void initParser(void);   /* needed in src/mal/mal.c */
-mal_export int parseMAL(Client cntxt, Symbol curPrg, int skipcomments);
+mal_export int parseMAL(Client cntxt, Symbol curPrg, int skipcomments, int 
lines);
 
 #endif /* _MAL_PARSER_H */
 
diff --git a/monetdb5/mal/mal_session.c b/monetdb5/mal/mal_session.c
--- a/monetdb5/mal/mal_session.c
+++ b/monetdb5/mal/mal_session.c
@@ -574,7 +574,7 @@ MALparser(Client c)
oldstate = *c->curprg->def;
 
prepareMalBlk(c->curprg->def, CURRENT(c));
-   if (parseMAL(c, c->curprg, 0) || c->curprg->def->errors) {
+   if (parseMAL(c, c->curprg, 0, INT_MAX) || c->curprg->def->errors) {
/* just complete it for visibility */
pushEndInstruction(c->curprg->def);
/* caught errors */
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: wlcr - Assertion off by one test solved

2017-01-05 Thread Martin Kersten

Changeset: 8b2051286081 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8b2051286081
Modified Files:
monetdb5/mal/mal_interpreter.c
Branch: wlcr
Log Message:

Assertion off by one test solved


diffs (12 lines):

diff --git a/monetdb5/mal/mal_interpreter.c b/monetdb5/mal/mal_interpreter.c
--- a/monetdb5/mal/mal_interpreter.c
+++ b/monetdb5/mal/mal_interpreter.c
@@ -1419,7 +1419,7 @@ void garbageCollector(Client cntxt, MalB
printStack(cntxt->fdout, mb, stk, 0);
}
 #endif
-   assert(mb->vtop < mb->vsize);
+   assert(mb->vtop <= mb->vsize);
(void) flag;
for (k = 0; k < mb->vtop; k++) {
//  if (isVarCleanup(mb, k) ){
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: wlcr - Merge with default

2017-01-05 Thread Martin Kersten

Changeset: c99a8a706b2a for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=c99a8a706b2a
Removed Files:
sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.sql
sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.err
sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out
Modified Files:
clients/Tests/MAL-signatures.stable.out
clients/Tests/MAL-signatures.stable.out.int128
clients/Tests/exports.stable.out
monetdb5/mal/mal.c
monetdb5/mal/mal_client.c
monetdb5/mal/mal_instruction.c
monetdb5/mal/mal_instruction.h
monetdb5/mal/mal_module.c
monetdb5/mal/mal_parser.c
monetdb5/mal/mal_session.c
monetdb5/modules/atoms/inet.c
sql/backends/monet5/mal_backend.c
sql/backends/monet5/rel_bin.c
sql/backends/monet5/sql.c
sql/backends/monet5/sql.h
sql/backends/monet5/sql_cat.c
sql/backends/monet5/sql_execute.c
sql/backends/monet5/sql_gencode.c
sql/backends/monet5/sql_gencode.h
sql/backends/monet5/sql_scenario.c
sql/backends/monet5/sql_statement.c
sql/backends/monet5/sql_statement.h
sql/backends/monet5/sqlcatalog.mal
sql/server/rel_psm.c
sql/server/rel_schema.c
sql/server/rel_sequence.c
sql/storage/store.c
sql/test/BugTracker-2009/Tests/All

sql/test/BugTracker-2010/Tests/error-truncated.Bug-2615.stable.err.Darwin

sql/test/BugTracker-2010/Tests/error-truncated.Bug-2615.stable.err.Windows

sql/test/BugTracker-2010/Tests/group-by_ordered_column.Bug-2564.stable.out.32bit
tools/mserver/shutdowntest.c
Branch: wlcr
Log Message:

Merge with default


diffs (truncated from 1692 to 300 lines):

diff --git a/clients/Tests/MAL-signatures.stable.out 
b/clients/Tests/MAL-signatures.stable.out
--- a/clients/Tests/MAL-signatures.stable.out
+++ b/clients/Tests/MAL-signatures.stable.out
@@ -8463,19 +8463,19 @@ Ready.
 [ "sqlblob",   "sqlblob",  "command sqlblob.sqlblob(s:str):sqlblob ",  
"BLOBsqlblob_fromstr;", ""  ]
 [ "sqlcatalog","alter_add_table",  "pattern 
sqlcatalog.alter_add_table(sname:str, mtnme:str, psnme:str, ptnme:str, 
action:int):void ", "SQLalter_add_table;",  "Catalog operation 
alter_add_table" ]
 [ "sqlcatalog","alter_del_table",  "pattern 
sqlcatalog.alter_del_table(sname:str, mtnme:str, psnme:str, ptnme:str, 
action:int):void ", "SQLalter_del_table;",  "Catalog operation 
alter_del_table" ]
-[ "sqlcatalog","alter_seq","pattern 
sqlcatalog.alter_seq(sname:str, seq:ptr, val:lng):void ",  "SQLalter_seq;", 
   "Catalog operation alter_seq"   ]
+[ "sqlcatalog","alter_seq","pattern 
sqlcatalog.alter_seq(sname:str, seqname:str, seq:ptr, val:lng):void ", 
"SQLalter_seq;","Catalog operation alter_seq"   ]
 [ "sqlcatalog","alter_set_table",  "pattern 
sqlcatalog.alter_set_table(sname:str, tnme:str, access:int):void ",
"SQLalter_set_table;",  "Catalog operation alter_set_table" ]
-[ "sqlcatalog","alter_table",  "pattern 
sqlcatalog.alter_table(sname:str, tbl:ptr, action:int):void ", 
"SQLalter_table;",  "Catalog operation alter_table" ]
+[ "sqlcatalog","alter_table",  "pattern 
sqlcatalog.alter_table(sname:str, tname:str, tbl:ptr, action:int):void ",  
"SQLalter_table;",  "Catalog operation alter_table" ]
 [ "sqlcatalog","alter_user",   "pattern 
sqlcatalog.alter_user(sname:str, passwrd:str, enc:int, schema:str, 
oldpasswrd:str):void ", "SQLalter_user;",   "Catalog operation 
alter_user"  ]
-[ "sqlcatalog","create_function",  "pattern 
sqlcatalog.create_function(sname:str, fcn:ptr):void ", "SQLcreate_function;",  
"Catalog operation create_function" ]
+[ "sqlcatalog","create_function",  "pattern 
sqlcatalog.create_function(sname:str, fname:str, fcn:ptr):void ",  
"SQLcreate_function;",  "Catalog operation create_function" ]
 [ "sqlcatalog","create_role",  "pattern 
sqlcatalog.create_role(sname:str, role:str, grator:int):void ",
"SQLcreate_role;",  "Catalog operation create_role" ]
 [ "sqlcatalog","create_schema","pattern 
sqlcatalog.create_schema(sname:str, auth:str, action:int):void ",  
"SQLcreate_schema;","Catalog operation create_schema"   ]
-[ "sqlcatalog","create_seq",   "pattern 
sqlcatalog.create_seq(sname:str, seq:ptr, action:int):void ",  
"SQLcreate_seq;",   "Catalog operation create_seq"  ]
-[ "sqlcatalog","create_table", "pattern 
sqlcatalog.create_table(sname:str, tbl:ptr, temp:int):void ",  
"SQLcreate_table;", "Catalog operation create_table"]
+[ "sqlcatalog","create_seq",   "pattern 
sqlcatalog.create_seq(sname:str, seqname:str, seq:ptr, action:int):void ", 
"SQLcreate_seq;",   "Catalog operation create_seq"  ]

MonetDB: wlcr - Save the state before patching MAL parser

2017-01-05 Thread Martin Kersten

Changeset: e58d856a620a for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e58d856a620a
Modified Files:
monetdb5/mal/mal_session.c
monetdb5/modules/mal/wlcr.c
sql/backends/monet5/Makefile.ag
sql/backends/monet5/sql_wlcr.c
sql/scripts/60_wlcr.sql
Branch: wlcr
Log Message:

Save the state before patching MAL parser


diffs (176 lines):

diff --git a/monetdb5/mal/mal_session.c b/monetdb5/mal/mal_session.c
--- a/monetdb5/mal/mal_session.c
+++ b/monetdb5/mal/mal_session.c
@@ -572,8 +572,8 @@ MALparser(Client c)
c->yycur = 0;
 
/* check for unfinished blocks */
-   if (c->blkmode)
-   return MAL_SUCCEED;
+   //if (c->blkmode)
+   //return MAL_SUCCEED;
/* empty files should be skipped as well */
if (c->curprg->def->stop == 1)
return MAL_SUCCEED;
@@ -618,8 +618,8 @@ MALengine(Client c)
MalBlkRecord oldstate = *c->curprg->def;
oldstate.stop = 0;
 
-   if (c->blkmode)
-   return MAL_SUCCEED;
+   //if (c->blkmode)
+   //return MAL_SUCCEED;
prg = c->curprg;
if (prg == NULL)
throw(SYNTAX, "mal.engine", SYNTAX_SIGNATURE);
diff --git a/monetdb5/modules/mal/wlcr.c b/monetdb5/modules/mal/wlcr.c
--- a/monetdb5/modules/mal/wlcr.c
+++ b/monetdb5/modules/mal/wlcr.c
@@ -117,7 +117,7 @@ WLCRmaster(Client cntxt, MalBlkPtr mb, M
if( fscanf(fd,"%d %d", _batch, _threshold) != 3)
fprintf(fd,"0 %d\n", wlcr_threshold);
fclose(fd);
-   mnstr_printf(cntxt->fdout,"#master wlcr_batch %d\n",wlcr_batch, 
wlcr_threshold);
+   mnstr_printf(cntxt->fdout,"#master wlcr_batch %d threshold 
%d\n",wlcr_batch, wlcr_threshold);
return MAL_SUCCEED;
 }
 
diff --git a/sql/backends/monet5/Makefile.ag b/sql/backends/monet5/Makefile.ag
--- a/sql/backends/monet5/Makefile.ag
+++ b/sql/backends/monet5/Makefile.ag
@@ -64,7 +64,7 @@ headers_mal = {
HEADERS = mal
DIR = libdir/monetdb5
SOURCES = sql_aggr_bte.mal  sql_aggr_flt.mal sql_aggr_dbl.mal  
sql_aggr_int.mal  sql_aggr_lng.mal \
-   sql_aggr_sht.mal  sql_decimal.mal  sql_inspect.mal \
+   sql_aggr_sht.mal  sql_decimal.mal  sql_inspect.mal sql_wlcr.mal\
sql_rank.mal sqlcatalog.mal sql_transaction.mal  sql.mal
 }
 
@@ -91,7 +91,7 @@ headers_autoload_hge = {
 EXTRA_DIST = 40_sql.mal \
41_sql_hge.mal sql_hge.mal sql_aggr_hge.mal sql_decimal_hge.mal \
sql_aggr_bte.mal  sql_aggr_flt.mal  sql_aggr_int.mal  sql_aggr_lng.mal \
-   sql_aggr_sht.mal  sql_decimal.mal  sql_inspect.mal \
+   sql_aggr_sht.mal  sql_decimal.mal  sql_inspect.mal sql_wlcr.mal \
sql_rank.mal sqlcatalog.mal sql_transaction.mal  sql.mal 
 
 EXTRA_DIST_DIR = Tests
diff --git a/sql/backends/monet5/sql_wlcr.c b/sql/backends/monet5/sql_wlcr.c
--- a/sql/backends/monet5/sql_wlcr.c
+++ b/sql/backends/monet5/sql_wlcr.c
@@ -6,6 +6,17 @@
  * Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
  */
 
+/*
+ * A master can be replicated by taking a binary copy of the 'bat' directory.
+ * This should be done under control of the program monetdb, e.g.
+ * monetdb replica  +
+ * 
+ * After restart of a mserver against the newly created image,
+ * the log files from the master are processed.
+ * Alternatively you start with an empty database.
+ *
+ * Since the wlcr files can be stored anywhere, the full path should be given.
+ */
 #include "monetdb_config.h"
 #include "sql.h"
 #include "wlcr.h"
@@ -13,23 +24,38 @@
 
 static str wlcr_master;
 static int wlcr_replaythreshold;
+static int wlcr_replaybatch;
 
 static str
 WLCRinit(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
 {
-int i = 1;
+int i = 1, j,k,l;
+   char path[PATHLENGTH];
+   FILE *fd;
 
(void) cntxt;
+   (void) k;
+   (void) l;
 
 wlcr_master = GDKgetenv("gdk_master");
 if (i< pci->argc+1 && getArgType(mb, pci, i) == TYPE_str){
 wlcr_master =  *getArgReference_str(stk,pci,i);
-wlcr_master = GDKfilepath(0,wlcr_master,"batch",0);
 i++;
 }
if( wlcr_master == NULL){
throw(SQL,"wlcr.init","Can not access the wlcr directory");
}
+   snprintf(path,PATHLENGTH,"%s%cwlcr", wlcr_master, DIR_SEP);
+   mnstr_printf(cntxt->fdout,"#Testing '%s'\n", path);
+   fd = fopen(path,"r");
+   if( fd == NULL){
+   throw(SQL,"wlcr.init","Can not access '%s'\n",path);
+   }
+   if( fscanf(fd,"%d %d %d", ,,) != 3){
+   throw(SQL,"wlcr.init","'%s' does not have proper number of 
arguments\n",path);
+   }
+   wlcr_replaybatch = j;
+
 if ( i < pci->argc+1 && getArgType(mb, pci, i) == TYPE_int){
 wlcr_replaythreshold = *getArgReference_int(stk,pci,i);
}
@@ -39,16 +65,42 @@ WLCRinit(Client cntxt, MalBlkPtr mb, Mal
 str
 WLCRreplay(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr

MonetDB: wlcr - Basic handling of WLCR logs writes

2017-01-05 Thread Martin Kersten

Changeset: f097aee982a1 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=f097aee982a1
Added Files:
sql/backends/monet5/sql_wlcr.c
sql/backends/monet5/sql_wlcr.h
sql/backends/monet5/sql_wlcr.mal
sql/scripts/60_wlcr.sql
Modified Files:
monetdb5/modules/mal/wlcr.c
monetdb5/modules/mal/wlcr.h
monetdb5/modules/mal/wlcr.mal
sql/backends/monet5/40_sql.mal
sql/backends/monet5/Makefile.ag
sql/scripts/Makefile.ag
Branch: wlcr
Log Message:

Basic handling of WLCR logs writes


diffs (truncated from 550 to 300 lines):

diff --git a/monetdb5/modules/mal/wlcr.c b/monetdb5/modules/mal/wlcr.c
--- a/monetdb5/modules/mal/wlcr.c
+++ b/monetdb5/modules/mal/wlcr.c
@@ -7,62 +7,129 @@
  */
 
 /*
- * (c) Martin Kersten
+ * (c) 2017 Martin Kersten
  * This module collects the workload-capture-replay statements during 
transaction execution. 
+ * It is used primarilly for replication management and workload replay
+ *
+ * The goal is to maintain a replica of a master database. All data of the 
master
+ * is basically only available for read only access. Accidental corruption of 
this
+ * data is avoided by setting ownership and access properties at the SQL level 
in the replica.
+ *
+ *
+ * IMPLEMENTATION
+ *
+ * The replica directory should be on a shared (global) file system.
+ * As default we use dbfarm/master.
+ *
+ * The binary dump for the database snapshot should be stored there in  
master/bat.
+ * The associated log files are stored as master/wlcr.
+ * Creation and restore of a snapshot should be a  monetdb option. TODO
+ *
+ * Replication management start when you run the command 
+ * CALL wlcr.master("(full)path to snapshot dir")
+ * It can also be passed as a command line parameter 
+ * --set wlcr_dir="(full)path to snapshot dir"
  *
  * Each wlcr log file contains a serial log for a transaction batch. 
- * Each job is identified by the original owner of the query, the snapshot 
identity (name+nr) against
- * which it was ran, an indication of the kind of transaction and 
commit/rollback, its runtime (in ms) and starting time.
+ * Each job is identified by the owner of the query, the snapshot tag,
+ * commit/rollback, its starting time and runtime (in ms).
  *
- * Replaying the wlcr against another server based on the same snapshot should 
produce a perfect copy.
- * Each job should be executed using the credentials of the user issuing the 
transaction.
- * Any failuer encountered terminates the replication process.
+ * Logging of queries can be further limited to those that satisfy a threshold.
+ * CALL wlcr.master("(full)path to snapshot dir", threshold)
+ * The threshold is given in milliseconds. A negative threshold leads to 
ignoring all queries.
+ *
+ * A replica server should issue the matching call
+ * CALL wlcr.synchronize("(full)path to snapshot dir")
+ *
+ * During synchronization only updates are executed for the user responsible 
for the call. 
+ * Queries are simply ignored unless needed as replacement for update actions.
+ *
+ * The alternative is to replay the log
+ * CALL wlcr.replay("(full)path to snapshot dir")
+ * In this mode all queries are executed under the credentials of the query 
owner, including those that lead to updates.
+ *
+ * Any failure encountered terminates the synchronization process, leaving a 
message in the merovingian log.
+ *
+ * The replay progress can be inspected using the function wlcr.drift() and 
wlcr.synced().
+ * The latter is true if all accessible log files have been processed.
  * 
- * All wlcr files should be stored on a shared file system for all replicas to 
access.
- * The default is a subdirectory of the database and act as a secondary 
database rebuild log.
- * The location can be overruled using a full path to a shared disk as 
GDKenvironemnt variable (wlcr_dir)
+ * The wlcr files purposely have a textual format derived from the MAL 
statements.
+ * It creates some overhead for copy into situations.
  *
- * The wlcr files have a textual format derived from the MAL statements.
- * This can be used to ease the implementation of the wlreplay
+ * The integrity of the wlcr directories is critical. For now we assume that 
all batches are available. 
+ * We should detect that wlcr.master() is issued after updates have taken 
place on the snapshot TODO.
  *
- * The logs may only be removed after a new snapshot has been taken or wlcr is 
disabled
  */
 #include "monetdb_config.h"
+#include 
 #include "mal_builder.h"
 #include "wlcr.h"
 
 static MT_Lock wlcr_lock MT_LOCK_INITIALIZER("wlcr_lock");
 
 
-int wlcr_duration = INT_MAX; // how long to capture default= 0
 int wlcr_threshold = 0; // threshold (milliseconds) for keeping readonly 
queries
-int wlcr_deltas = 1;  // sent the delta values
-int wlcr_all = 1;  // also ship failed transaction
-str wlcr_snapshot= "baseline"; // name assigned to the snapshot
-int wlcr_unit = 0; // last job executed
+str

MonetDB: wlcr - merge with default

2017-01-05 Thread Martin Kersten

Changeset: 9d358b84f3c6 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9d358b84f3c6
Added Files:
sql/test/BugTracker-2017/Tests/All
sql/test/BugTracker-2017/Tests/avggroupbysq.Bug-6178.sql
sql/test/BugTracker-2017/Tests/avggroupbysq.Bug-6178.stable.err
sql/test/BugTracker-2017/Tests/avggroupbysq.Bug-6178.stable.out
sql/test/BugTracker-2017/Tests/shutdown.Bug-6182.sql
sql/test/BugTracker-2017/Tests/shutdown.Bug-6182.stable.err
sql/test/BugTracker-2017/Tests/shutdown.Bug-6182.stable.out
Modified Files:
clients/Tests/MAL-signatures.stable.out
clients/Tests/MAL-signatures.stable.out.int128
clients/Tests/exports.stable.out
gdk/ChangeLog
gdk/gdk.h
gdk/gdk_aggr.c
gdk/gdk_atoms.c
gdk/gdk_bat.c
gdk/gdk_bbp.c
gdk/gdk_bbp.h
gdk/gdk_interprocess.c
gdk/gdk_interprocess.h
gdk/gdk_logger.c
gdk/gdk_tm.c
gdk/gdk_value.c
monetdb5/mal/mal.h
monetdb5/mal/mal_client.c
monetdb5/mal/mal_debugger.c
monetdb5/mal/mal_factory.c
monetdb5/mal/mal_interpreter.c
monetdb5/mal/mal_linker.c
monetdb5/mal/mal_linker.h
monetdb5/mal/mal_parser.c
monetdb5/mal/mal_resolve.c
monetdb5/modules/kernel/aggr.c
monetdb5/modules/mal/clients.c
monetdb5/modules/mal/groupby.c
monetdb5/modules/mal/mal_mapi.c
monetdb5/modules/mal/orderidx.c
monetdb5/modules/mal/tablet.c
monetdb5/modules/mal/tokenizer.c
monetdb5/optimizer/opt_mergetable.c
monetdb5/optimizer/opt_statistics.c
monetdb5/scheduler/run_memo.c
sql/backends/monet5/UDF/pyapi/connection.c
sql/backends/monet5/UDF/pyapi/conversion.c
sql/backends/monet5/UDF/pyapi/conversion.h
sql/backends/monet5/UDF/pyapi/emit.c
sql/backends/monet5/UDF/pyapi/formatinput.c
sql/backends/monet5/UDF/pyapi/pyapi.c
sql/backends/monet5/UDF/pyapi/pyapi.h
sql/backends/monet5/UDF/pyapi/pyheader.h
sql/backends/monet5/UDF/pyapi/pyloader.c
sql/backends/monet5/UDF/pyapi/pytypes.c
sql/backends/monet5/UDF/pyapi/type_conversion.c
sql/backends/monet5/UDF/pyapi/unicode.c
sql/backends/monet5/rel_bin.c
sql/backends/monet5/sql.c
sql/backends/monet5/sql_gencode.c
sql/backends/monet5/sql_statement.c
sql/backends/monet5/vaults/lidar/lidar.c
sql/backends/monet5/vaults/netcdf/netcdf.c
sql/backends/monet5/vaults/shp/shp.c
sql/common/sql_mem.c
sql/include/sql_mem.h
sql/server/rel_optimizer.c
sql/server/rel_psm.c
sql/server/sql_mvc.c
sql/server/sql_parser.y
sql/storage/bat/bat_utils.c
sql/storage/bat/res_table.c

sql/test/BugTracker/Tests/groupby_orderby_nonselected.SF-1723863.stable.err
sql/test/BugTracker/Tests/subquery_orderby.SF-1688915.stable.err
sql/test/BugTracker/Tests/subquery_orderby.SF-1688915.stable.out
testing/Mtest.py.in
tools/mserver/shutdowntest.c
Branch: wlcr
Log Message:

merge with default


diffs (truncated from 2622 to 300 lines):

diff --git a/clients/Tests/MAL-signatures.stable.out 
b/clients/Tests/MAL-signatures.stable.out
--- a/clients/Tests/MAL-signatures.stable.out
+++ b/clients/Tests/MAL-signatures.stable.out
@@ -8309,7 +8309,7 @@ Ready.
 [ "sql",   "analyze",  "pattern sql.analyze(minmax:int, sample:lng, 
sch:str):void ",   "sql_analyze;", ""  ]
 [ "sql",   "analyze",  "pattern sql.analyze(minmax:int, sample:lng, 
sch:str, tbl:str):void ",  "sql_analyze;", ""  ]
 [ "sql",   "analyze",  "pattern sql.analyze(minmax:int, sample:lng, 
sch:str, tbl:str, col:str):void ", "sql_analyze;", "Update the database 
statistics table"  ]
-[ "sql",   "append",   "pattern sql.append(mvc:int, sname:str, 
tname:str, cname:str, ins:any):int ",   "mvc_append_wrap;", "Append b to 
the column tname.cname (possibly optimized to replace the insert bat of 
tname.cname (returns sequence number for order dependence)"]
+[ "sql",   "append",   "pattern sql.append(mvc:int, sname:str, 
tname:str, cname:str, ins:any):int ",   "mvc_append_wrap;", "Append to the 
column tname.cname (possibly optimized to replace the insert bat of 
tname.cname. Returns sequence number for order dependence."  ]
 [ "sql",   "argRecord","pattern sql.argRecord():str ", 
"SQLargRecord;","Glue together the calling sequence"]
 [ "sql",   "argRecord","pattern sql.argRecord(a:any...):str ", 
"SQLargRecord;","Glue together the calling sequence"]
 [ "sql",   "assert",   "pattern sql.assert(b:bit, msg:str):void ", 
"SQLassert;",   "Generate an exception when b==true"]
@@ -8323,7 +8323,7 @@ Ready.
 [ "sql",   "bind_idxbat",  "pattern sql.bind_idxbat(mvc:int, schema:str,

MonetDB: wlcr - WorkLoad Capture and Replay

2017-01-05 Thread Martin Kersten

Changeset: 3bf5307afae1 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3bf5307afae1
Added Files:
monetdb5/modules/mal/wlcr.c
monetdb5/modules/mal/wlcr.h
monetdb5/modules/mal/wlcr.mal
monetdb5/optimizer/opt_wlcr.c
monetdb5/optimizer/opt_wlcr.h
Modified Files:
monetdb5/mal/mal_builder.c
monetdb5/mal/mal_client.c
monetdb5/mal/mal_client.h
monetdb5/mal/mal_interpreter.c
monetdb5/modules/mal/Makefile.ag
monetdb5/modules/mal/mal_init.mal
monetdb5/optimizer/Makefile.ag
monetdb5/optimizer/opt_pipes.c
monetdb5/optimizer/opt_prelude.c
monetdb5/optimizer/opt_prelude.h
monetdb5/optimizer/opt_support.c
monetdb5/optimizer/opt_wrapper.c
monetdb5/optimizer/optimizer.mal
sql/backends/monet5/sql.c
sql/backends/monet5/sql_scenario.c
sql/backends/monet5/sql_transaction.c
Branch: wlcr
Log Message:

WorkLoad Capture and Replay
First batch of extensions. See wlcr.c for more info


diffs (truncated from 1215 to 300 lines):

diff --git a/monetdb5/mal/mal_builder.c b/monetdb5/mal/mal_builder.c
--- a/monetdb5/mal/mal_builder.c
+++ b/monetdb5/mal/mal_builder.c
@@ -454,11 +454,16 @@ pushStr(MalBlkPtr mb, InstrPtr q, const 
if (q == NULL)
return NULL;
cst.vtype= TYPE_str;
-   if ((cst.val.sval= GDKstrdup(Val)) == NULL) {
-   freeInstruction(q);
-   return NULL;
+   if( Val == 0){
+   cst.val.sval = 0;
+   cst.len= 0;
+   } else{
+   if ( Val != NULL && (cst.val.sval= GDKstrdup(Val)) == NULL) {
+   freeInstruction(q);
+   return NULL;
+   }
+   cst.len= (int) strlen(cst.val.sval);
}
-   cst.len= (int) strlen(cst.val.sval);
_t = defConstant(mb,TYPE_str,);
return pushArgument(mb, q, _t);
 }
diff --git a/monetdb5/mal/mal_client.c b/monetdb5/mal/mal_client.c
--- a/monetdb5/mal/mal_client.c
+++ b/monetdb5/mal/mal_client.c
@@ -251,6 +251,8 @@ MCinitClientRecord(Client c, oid user, b
/* create a recycler cache */
c->exception_buf_initialized = 0;
c->error_row = c->error_fld = c->error_msg = c->error_input = NULL;
+   c->wlcr_kind = 0;
+   c->wlcr = NULL;
 #ifndef HAVE_EMBEDDED /* no authentication in embedded mode */
{
str msg = AUTHgetUsername(>username, c);
@@ -403,6 +405,10 @@ freeClient(Client c)
BBPdecref(c->error_msg->batCacheid,TRUE);
BBPdecref(c->error_input->batCacheid,TRUE);
c->error_row = c->error_fld = c->error_msg = c->error_input = 
NULL;
+   if( c->wlcr)
+   freeMalBlk(c->wlcr);
+   c->wlcr_kind = 0;
+   c->wlcr = NULL;
}
if (t)
THRdel(t);  /* you may perform suicide */
diff --git a/monetdb5/mal/mal_client.h b/monetdb5/mal/mal_client.h
--- a/monetdb5/mal/mal_client.h
+++ b/monetdb5/mal/mal_client.h
@@ -171,6 +171,11 @@ typedef struct CLIENT {
 */
bit active; /* processing a query or not */
Workset inprogress[THREADS];
+   /*
+* The workload for replication/replay is saved initially as a MAL 
block.
+*/
+   int wlcr_kind;  
+   MalBlkPtr wlcr;
/*  
 *  Errors during copy into are collected in a user specific column 
set
 */
diff --git a/monetdb5/mal/mal_interpreter.c b/monetdb5/mal/mal_interpreter.c
--- a/monetdb5/mal/mal_interpreter.c
+++ b/monetdb5/mal/mal_interpreter.c
@@ -1434,6 +1434,7 @@ void garbageCollector(Client cntxt, MalB
printStack(cntxt->fdout, mb, stk, 0);
}
 #endif
+   assert(mb->vtop < mb->vsize);
(void) flag;
for (k = 0; k < mb->vtop; k++) {
//  if (isVarCleanup(mb, k) ){
diff --git a/monetdb5/modules/mal/Makefile.ag b/monetdb5/modules/mal/Makefile.ag
--- a/monetdb5/modules/mal/Makefile.ag
+++ b/monetdb5/modules/mal/Makefile.ag
@@ -36,6 +36,7 @@ lib_mal = {
mkey.c mkey.h \
manifold.c manifold.h \
oltp.c oltp.h \
+   wlcr.c wlcr.h \
pcre.c \
profiler.c profiler.h \
querylog.c querylog.h \
@@ -59,7 +60,7 @@ headers_mal = {
inspect.mal manual.mal mal_io.mal mkey.mal manifold.mal \
iterator.mal clients.mal \
factories.mal groupby.mal mdb.mal pcre.mal mat.mal \
-   transaction.mal oltp.mal \
+   transaction.mal oltp.mal wlcr.mal \
mal_mapi.mal sabaoth.mal remote.mal  \
txtsim.mal \
tokenizer.mal sample.mal json_util.mal \
diff --git a/monetdb5/modules/mal/mal_init.mal 
b/monetdb5/modules/mal/mal_init.mal
--- a/monetdb5/modules/mal/mal_init.mal
+++

monetdb-java: embedded - Made fixings regarding the null values ...

2017-01-05 Thread Pedro Ferreira

Changeset: 6f74e01c57da for monetdb-java
URL: http://dev.monetdb.org/hg/monetdb-java?cmd=changeset;node=6f74e01c57da
Added Files:
src/main/java/nl/cwi/monetdb/jdbc/MonetINET.java
Removed Files:
src/main/java/nl/cwi/monetdb/jdbc/types/MonetINET.java
src/main/java/nl/cwi/monetdb/jdbc/types/MonetURL.java
Modified Files:
build.xml
example/MJDBCTest.java
example/PreparedExample.java
example/SQLImport.java
example/SQLcopyinto.java
src/main/java/nl/cwi/monetdb/client/JMonetDB.java
src/main/java/nl/cwi/monetdb/client/JdbcClient.java
src/main/java/nl/cwi/monetdb/jdbc/MonetBlob.java
src/main/java/nl/cwi/monetdb/jdbc/MonetClob.java
src/main/java/nl/cwi/monetdb/jdbc/MonetConnection.java
src/main/java/nl/cwi/monetdb/jdbc/MonetDataSource.java
src/main/java/nl/cwi/monetdb/jdbc/MonetDatabaseMetaData.java
src/main/java/nl/cwi/monetdb/jdbc/MonetDriver.java.in
src/main/java/nl/cwi/monetdb/jdbc/MonetPreparedStatement.java
src/main/java/nl/cwi/monetdb/jdbc/MonetResultSet.java
src/main/java/nl/cwi/monetdb/jdbc/MonetSavepoint.java
src/main/java/nl/cwi/monetdb/jdbc/MonetStatement.java
src/main/java/nl/cwi/monetdb/jdbc/MonetWrapper.java
src/main/java/nl/cwi/monetdb/mcl/connection/ControlCommands.java
src/main/java/nl/cwi/monetdb/mcl/connection/IMonetDBLanguage.java
src/main/java/nl/cwi/monetdb/mcl/connection/MCLException.java
src/main/java/nl/cwi/monetdb/mcl/connection/SenderThread.java

src/main/java/nl/cwi/monetdb/mcl/connection/helpers/BufferReallocator.java
src/main/java/nl/cwi/monetdb/mcl/connection/helpers/ChannelSecurity.java

src/main/java/nl/cwi/monetdb/mcl/connection/helpers/GregorianCalendarParser.java
src/main/java/nl/cwi/monetdb/mcl/connection/mapi/AbstractSocket.java
src/main/java/nl/cwi/monetdb/mcl/connection/mapi/MapiConnection.java
src/main/java/nl/cwi/monetdb/mcl/connection/mapi/MapiLanguage.java
src/main/java/nl/cwi/monetdb/mcl/connection/mapi/OldMapiSocket.java
src/main/java/nl/cwi/monetdb/mcl/protocol/AbstractProtocol.java
src/main/java/nl/cwi/monetdb/mcl/protocol/ProtocolException.java
src/main/java/nl/cwi/monetdb/mcl/protocol/ServerResponses.java
src/main/java/nl/cwi/monetdb/mcl/protocol/StarterHeaders.java
src/main/java/nl/cwi/monetdb/mcl/protocol/TableResultHeaders.java
src/main/java/nl/cwi/monetdb/mcl/protocol/newmapi/NewMapiProtocol.java
src/main/java/nl/cwi/monetdb/mcl/protocol/oldmapi/OldMapiProtocol.java

src/main/java/nl/cwi/monetdb/mcl/protocol/oldmapi/OldMapiServerResponseParser.java

src/main/java/nl/cwi/monetdb/mcl/protocol/oldmapi/OldMapiStartOfHeaderParser.java

src/main/java/nl/cwi/monetdb/mcl/protocol/oldmapi/OldMapiTableHeaderParser.java

src/main/java/nl/cwi/monetdb/mcl/protocol/oldmapi/OldMapiTupleLineParser.java
src/main/java/nl/cwi/monetdb/mcl/responses/AutoCommitResponse.java
src/main/java/nl/cwi/monetdb/mcl/responses/DataBlockResponse.java
src/main/java/nl/cwi/monetdb/mcl/responses/IIncompleteResponse.java
src/main/java/nl/cwi/monetdb/mcl/responses/IResponse.java
src/main/java/nl/cwi/monetdb/mcl/responses/ResultSetResponse.java
src/main/java/nl/cwi/monetdb/mcl/responses/SchemaResponse.java
src/main/java/nl/cwi/monetdb/mcl/responses/UpdateResponse.java
src/main/java/nl/cwi/monetdb/merovingian/Control.java
src/main/java/nl/cwi/monetdb/merovingian/MerovingianException.java
src/main/java/nl/cwi/monetdb/merovingian/SabaothDB.java
src/main/java/nl/cwi/monetdb/util/CmdLineOpts.java
src/main/java/nl/cwi/monetdb/util/Exporter.java
src/main/java/nl/cwi/monetdb/util/Extract.java
src/main/java/nl/cwi/monetdb/util/OptionsException.java
src/main/java/nl/cwi/monetdb/util/SQLExporter.java
src/main/java/nl/cwi/monetdb/util/SQLRestore.java
src/main/java/nl/cwi/monetdb/util/XMLExporter.java
tests/BugConcurrent_clients_SF_1504657.java
tests/BugConcurrent_sequences.java
tests/BugDatabaseMetaData_Bug_3356.java
tests/BugDecimalRound_Bug_3561.java
tests/BugExecuteUpdate_Bug_3350.java
tests/BugSetQueryTimeout_Bug_3357.java
tests/Test_Cautocommit.java
tests/Test_Cforkbomb.java
tests/Test_CisValid.java
tests/Test_Clargequery.java
tests/Test_Cmanycon.java
tests/Test_Creplysize.java
tests/Test_Csavepoints.java
tests/Test_Csendthread.java
tests/Test_Ctransaction.java
tests/Test_Dobjects.java
tests/Test_FetchSize.java
tests/Test_Int128.java
tests/Test_PSgeneratedkeys.java
tests/Test_PSgetObject.java
tests/Test_PSlargeamount.java
tests/Test_PSlargebatchval.java

MonetDB: Dec2016 - Adding known referential integrity check test...

2017-01-05 Thread Martin van Dinther

Changeset: fe679689c934 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fe679689c934
Added Files:
sql/test/sys-schema/Tests/check_ForeignKey_referential_integrity.sql

sql/test/sys-schema/Tests/check_ForeignKey_referential_integrity.stable.err

sql/test/sys-schema/Tests/check_ForeignKey_referential_integrity.stable.out
Modified Files:
sql/test/sys-schema/Tests/All
Branch: Dec2016
Log Message:

Adding known referential integrity check test script for sys (and tmp) tables.
The foreign keys in sys.dependencies are complex as they depend on the value of 
depend_type and may refer to multiple table per column!
This requires more analysis before they can be defined and added. See TODO 
comments.


diffs (truncated from 593 to 300 lines):

diff --git a/sql/test/sys-schema/Tests/All b/sql/test/sys-schema/Tests/All
--- a/sql/test/sys-schema/Tests/All
+++ b/sql/test/sys-schema/Tests/All
@@ -1,2 +1,3 @@
 check_PrimaryKey_uniqueness
 check_AlternateKey_uniqueness
+check_ForeignKey_referential_integrity
diff --git 
a/sql/test/sys-schema/Tests/check_ForeignKey_referential_integrity.sql 
b/sql/test/sys-schema/Tests/check_ForeignKey_referential_integrity.sql
new file mode 100644
--- /dev/null
+++ b/sql/test/sys-schema/Tests/check_ForeignKey_referential_integrity.sql
@@ -0,0 +1,123 @@
+-- check all standard sys (and tmp) tables on Referential integrity
+-- All queries should return NO rows (so no invalid references found).
+SELECT * FROM sys.schemas WHERE authorization NOT IN (SELECT id FROM 
sys.auths);
+SELECT * FROM sys.schemas WHERE owner NOT IN (SELECT id FROM sys.auths);
+
+SELECT * FROM sys.tables WHERE schema_id NOT IN (SELECT id FROM sys.schemas);
+SELECT * FROM sys._tables WHERE schema_id NOT IN (SELECT id FROM sys.schemas);
+SELECT * FROM tmp._tables WHERE schema_id NOT IN (SELECT id FROM sys.schemas);
+SELECT * FROM sys.tables WHERE type NOT IN (SELECT table_type_id FROM 
sys.table_types);
+SELECT * FROM sys._tables WHERE type NOT IN (SELECT table_type_id FROM 
sys.table_types);
+SELECT * FROM tmp._tables WHERE type NOT IN (SELECT table_type_id FROM 
sys.table_types);
+
+SELECT * FROM sys.columns WHERE table_id NOT IN (SELECT id FROM sys.tables);
+SELECT * FROM sys._columns WHERE table_id NOT IN (SELECT id FROM sys._tables);
+SELECT * FROM tmp._columns WHERE table_id NOT IN (SELECT id FROM tmp._tables);
+SELECT * FROM sys.columns WHERE type NOT IN (SELECT sqlname FROM sys.types);
+SELECT * FROM sys._columns WHERE type NOT IN (SELECT sqlname FROM sys.types);
+SELECT * FROM sys._columns WHERE type NOT IN (SELECT sqlname FROM sys.types);
+
+SELECT * FROM sys.functions WHERE schema_id NOT IN (SELECT id FROM 
sys.schemas);
+-- SELECT * FROM sys.functions WHERE type NOT IN (SELECT id FROM 
sys.function_types);  -- table sys.function_types does not yet exist
+SELECT * FROM sys.functions WHERE type NOT IN (1,2,3,4,5,6,7);  -- replace 
this check when table sys.function_types becomes available
+-- SELECT * FROM sys.functions WHERE language NOT IN (SELECT language_id FROM 
sys.function_languages);  -- table sys.function_languages does not yet exist
+SELECT * FROM sys.functions WHERE language NOT IN (0,1,2,3,4,5,6,7);  -- 
replace this check when table sys.function_languages becomes available
+
+SELECT * FROM sys.systemfunctions WHERE function_id NOT IN (SELECT id FROM 
sys.functions);
+-- systemfunctions should refer only to functions in MonetDB system schemas 
(on Dec2016 these are: sys, json, profiler and bam)
+SELECT * FROM sys.systemfunctions WHERE function_id NOT IN (SELECT id FROM 
sys.functions WHERE schema_id IN (SELECT id FROM sys.schemas WHERE name IN 
('sys','json','profiler','bam')));
+
+SELECT * FROM sys.args WHERE func_id NOT IN (SELECT id FROM sys.functions);
+SELECT * FROM sys.args WHERE type NOT IN (SELECT sqlname FROM sys.types);
+
+SELECT * FROM sys.types WHERE schema_id NOT IN (SELECT id FROM sys.schemas);
+SELECT * FROM sys.types WHERE schema_id NOT IN (SELECT id FROM sys.schemas) 
AND schema_id <> 0;
+
+SELECT * FROM sys.keys WHERE table_id NOT IN (SELECT id FROM sys.tables);
+SELECT * FROM sys.keys WHERE table_id NOT IN (SELECT id FROM sys._tables);
+SELECT * FROM tmp.keys WHERE table_id NOT IN (SELECT id FROM tmp._tables);
+-- SELECT * FROM sys.keys WHERE type NOT IN (SELECT key_type_id FROM 
sys.key_types);  -- table sys.key_types does not yet exist
+-- SELECT * FROM tmp.keys WHERE type NOT IN (SELECT key_type_id FROM 
sys.key_types);  -- table sys.key_types does not yet exist
+SELECT * FROM sys.keys WHERE type NOT IN (0, 1, 2);  -- replace this check 
when table sys.key_types becomes available
+SELECT * FROM tmp.keys WHERE type NOT IN (0, 1, 2);  -- replace this check 
when table sys.key_types becomes available
+
+SELECT * FROM sys.idxs WHERE table_id NOT IN (SELECT id FROM sys.tables);
+SELECT * FROM sys.idxs WHERE table_id NOT IN (SELECT id FROM sys._tables);
+SELECT * FROM tmp.idxs WHERE table_id NOT IN (SELECT id FROM tmp._tables);

MonetDB: default - Merge with Dec2016 branch.

2017-01-05 Thread Sjoerd Mullender

Changeset: 940183ddc2a0 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=940183ddc2a0
Added Files:
README.rst
Removed Files:
README
Modified Files:
HowToStart.rst
Makefile.ag
monetdb5/extras/rapi/Tests/rapi08.malC
sql/backends/monet5/vaults/bam/bam_wrapper.c
sql/server/sql_parser.y
Branch: default
Log Message:

Merge with Dec2016 branch.


diffs (220 lines):

diff --git a/HowToStart.rst b/HowToStart.rst
--- a/HowToStart.rst
+++ b/HowToStart.rst
@@ -24,7 +24,7 @@ This document assumes that you are plann
 installing MonetDB on a Unix-like system (e.g., Linux, IRIX, Solaris,
 AIX, Mac OS X/Darwin, or CYGWIN).  For compilation and installation on
 a native Windows system (NT, 2000, XP) see the instructions in the
-file `../buildtools/doc/windowsbuild.rst`__.
+file `buildtools/doc/windowsbuild.rst`__.
 
 __ http://dev.monetdb.org/downloads/
 __ Windows-Installation.html
diff --git a/Makefile.ag b/Makefile.ag
--- a/Makefile.ag
+++ b/Makefile.ag
@@ -13,7 +13,7 @@ SUBDIRS = buildtools common clients \
HAVE_TESTING?testing \
 
 EXTRA_DIST = bootstrap configure configure.ac configure.ag libversions \
-MonetDB.spec rpm.mk.in COPYING README license.txt 
HowToStart.rst
+MonetDB.spec rpm.mk.in COPYING README.rst license.txt 
HowToStart.rst
 
 EXTRA_DIST_DIR = NT debian MacOSX
 
diff --git a/README b/README
deleted file mode 100644
--- a/README
+++ /dev/null
@@ -1,26 +0,0 @@
-The MonetDB Database System
-===
-
-The MonetDB database is developed by the CWI database research group
-(see http://www.monetdb.org/).
-
-Via the MonetDB project we have brought the MonetDB system in open source,
-where it is accessible at http://www.monetdb.org/Downloads/
-
-The MonetDB database system is a high-performance database kernel for
-query-intensive applications. The MonetDB kernel works together with an
-SQL frontend that is in a separate CVS module.
-
-If you got a source distribution, please compile and install MonetDB first,
-following the instructions in the file 'HowToStart.rst' (for Unix)
-or 'buildtools/doc/windowsbuild.rst' (for Windows).
-
-
-Copyright Notice
-
-
-This Source Code Form is subject to the terms of the Mozilla Public
-License, v. 2.0.  If a copy of the MPL was not distributed with this
-file, You can obtain one at http://mozilla.org/MPL/2.0/.
-
-Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
diff --git a/README.rst b/README.rst
new file mode 100644
--- /dev/null
+++ b/README.rst
@@ -0,0 +1,34 @@
+The MonetDB Database System
+===
+
+The MonetDB database is developed by the CWI database research group
+(see http://www.monetdb.org/).
+
+Via the MonetDB project we have brought the MonetDB system in open source,
+where it is accessible at http://www.monetdb.org/Downloads/
+
+The MonetDB database system is a high-performance database kernel for
+query-intensive applications. The MonetDB source can be found at our `Mercurial
+server`__. There is also a `github mirror`__ that is updated once a day.
+
+.. _MonetDB: http://dev.monetdb.org/hg/MonetDB/
+__ MonetDB_
+
+.. _github: https://github.com/MonetDB/MonetDB
+__ github_
+
+If you got a source distribution, please compile and install MonetDB first,
+following the instructions in the file `HowToStart.rst`__ (for Unix)
+or `buildtools/doc/windowsbuild.rst`__ (for Windows).
+
+__ HowToStart.rst
+__ buildtools/doc/windowsbuild.rst
+
+Copyright Notice
+
+
+This Source Code Form is subject to the terms of the Mozilla Public
+License, v. 2.0.  If a copy of the MPL was not distributed with this
+file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
diff --git a/sql/backends/monet5/vaults/bam/bam_wrapper.c 
b/sql/backends/monet5/vaults/bam/bam_wrapper.c
--- a/sql/backends/monet5/vaults/bam/bam_wrapper.c
+++ b/sql/backends/monet5/vaults/bam/bam_wrapper.c
@@ -1613,13 +1613,16 @@ bam1_t2alignment(bam_wrapper * bw, lng v
a_out->cigar[1] = '\0';
} else {
uint32_t *cigar_bin = bam1_cigar(a_in);
+   uint32_t c;
int index = 0;
 
for (i = 0; i < a_in->core.n_cigar; ++i) {
+   /* work around unaligned access */
+   memcpy(, _bin[i], sizeof(c));
snprintf(_out->cigar[index],
 a_out->cigar_size - index, "%u%c",
-cigar_bin[i] >> BAM_CIGAR_SHIFT,
-bam_cigar_opchr(cigar_bin[i]));
+c >> BAM_CIGAR_SHIFT,
+bam_cigar_opchr(c));
index += strlen(_out->cigar[index]);
}
}
@@ -1750,25 +1753,37 @@ write_aux_bam1_t(bam_wrapper * bw, bam1_

MonetDB: Dec2016 - Don't shift with a negative amount or with to...

2017-01-05 Thread Sjoerd Mullender

Changeset: 5eca89f26310 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=5eca89f26310
Modified Files:
sql/server/sql_parser.y
Branch: Dec2016
Log Message:

Don't shift with a negative amount or with too many places.


diffs (20 lines):

diff --git a/sql/server/sql_parser.y b/sql/server/sql_parser.y
--- a/sql/server/sql_parser.y
+++ b/sql/server/sql_parser.y
@@ -4387,10 +4387,12 @@ literal:
  if (!err) {
int bits = digits2bits(digits), obits = bits;
 
-   for (;(one<<(bits-1)) > value; bits--)
-   ;
-  
-   if (bits != obits && 
+   while (bits > 0 &&
+  (bits == sizeof(value) * 8 ||
+   (one << (bits - 1)) > value))
+ bits--;
+
+   if (bits != obits &&
   (bits == 8 || bits == 16 || bits == 32 || bits == 64))
bits++;

___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Dec2016 - No unaligned memory access.

2017-01-05 Thread Sjoerd Mullender

Changeset: 64670d48b9af for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=64670d48b9af
Modified Files:
sql/backends/monet5/vaults/bam/bam_wrapper.c
Branch: Dec2016
Log Message:

No unaligned memory access.


diffs (106 lines):

diff --git a/sql/backends/monet5/vaults/bam/bam_wrapper.c 
b/sql/backends/monet5/vaults/bam/bam_wrapper.c
--- a/sql/backends/monet5/vaults/bam/bam_wrapper.c
+++ b/sql/backends/monet5/vaults/bam/bam_wrapper.c
@@ -1613,13 +1613,16 @@ bam1_t2alignment(bam_wrapper * bw, lng v
a_out->cigar[1] = '\0';
} else {
uint32_t *cigar_bin = bam1_cigar(a_in);
+   uint32_t c;
int index = 0;
 
for (i = 0; i < a_in->core.n_cigar; ++i) {
+   /* work around unaligned access */
+   memcpy(, _bin[i], sizeof(c));
snprintf(_out->cigar[index],
 a_out->cigar_size - index, "%u%c",
-cigar_bin[i] >> BAM_CIGAR_SHIFT,
-bam_cigar_opchr(cigar_bin[i]));
+c >> BAM_CIGAR_SHIFT,
+bam_cigar_opchr(c));
index += strlen(_out->cigar[index]);
}
}
@@ -1750,25 +1753,37 @@ write_aux_bam1_t(bam_wrapper * bw, bam1_
kputw(*(int8_t *) s, _value_stream);
++s;
} else if (type == 'S') {
+   uint16_t u;
+   memcpy(, s, sizeof(uint16_t));
type_str[0] = 'i';
-   kputw(*(uint16_t *) s, _value_stream);
+   kputw(u, _value_stream);
s += 2;
} else if (type == 's') {
+   int16_t i;
+   memcpy(, s, sizeof(int16_t));
type_str[0] = 'i';
-   kputw(*(int16_t *) s, _value_stream);
+   kputw(i, _value_stream);
s += 2;
} else if (type == 'I') {
+   uint32_t u;
+   memcpy(, s, sizeof(uint32_t));
type_str[0] = 'i';
-   kputuw(*(uint32_t *) s, _value_stream);
+   kputuw(u, _value_stream);
s += 4;
} else if (type == 'i') {
-   kputw(*(int32_t *) s, _value_stream);
+   int32_t i;
+   memcpy(, s, sizeof(int32_t));
+   kputw(i, _value_stream);
s += 4;
} else if (type == 'f') {
-   ksprintf(_value_stream, "%g", *(float *) s);
+   float f;
+   memcpy(, s, sizeof(float));
+   ksprintf(_value_stream, "%g", f);
s += 4;
} else if (type == 'd') {
-   ksprintf(_value_stream, "%lg", *(double *) s);
+   double d;
+   memcpy(, s, sizeof(double));
+   ksprintf(_value_stream, "%lg", d);
s += 8;
} else if (type == 'Z' || type == 'H') {
while (*s) {
@@ -1794,24 +1809,29 @@ write_aux_bam1_t(bam_wrapper * bw, bam1_
  _value_stream);
++s;
} else if ('s' == sub_type) {
-   kputw(*(int16_t *) s,
- _value_stream);
+   int16_t i;
+   memcpy(, s, sizeof(int16_t));
+   kputw(i, _value_stream);
s += 2;
} else if ('S' == sub_type) {
-   kputw(*(uint16_t *) s,
- _value_stream);
+   uint16_t u;
+   memcpy(, s, sizeof(uint16_t));
+   kputw(u, _value_stream);
s += 2;
} else if ('i' == sub_type) {
-   kputw(*(int32_t *) s,
- _value_stream);
+   int32_t i;
+   memcpy(, s, sizeof(int32_t));
+   kputw(i, _value_stream);
s += 4;
} else if ('I' == sub_type) {
-   kputuw(*(uint32_t *) s,
-

MonetDB: Dec2016 - README was converted to restructured text.

2017-01-05 Thread Sjoerd Mullender

Changeset: aa45e0eb6cd2 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=aa45e0eb6cd2
Modified Files:
Makefile.ag
Branch: Dec2016
Log Message:

README was converted to restructured text.


diffs (12 lines):

diff --git a/Makefile.ag b/Makefile.ag
--- a/Makefile.ag
+++ b/Makefile.ag
@@ -13,7 +13,7 @@ SUBDIRS = buildtools common clients \
HAVE_TESTING?testing \
 
 EXTRA_DIST = bootstrap configure configure.ac configure.ag libversions \
-MonetDB.spec rpm.mk.in COPYING README license.txt 
HowToStart.rst
+MonetDB.spec rpm.mk.in COPYING README.rst license.txt 
HowToStart.rst
 
 EXTRA_DIST_DIR = NT debian MacOSX
 
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Dec2016 - Backported test from default branch.

2017-01-05 Thread Sjoerd Mullender

Changeset: d309735fc70d for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=d309735fc70d
Modified Files:
monetdb5/extras/rapi/Tests/rapi08.malC
Branch: Dec2016
Log Message:

Backported test from default branch.


diffs (50 lines):

diff --git a/monetdb5/extras/rapi/Tests/rapi08.malC 
b/monetdb5/extras/rapi/Tests/rapi08.malC
--- a/monetdb5/extras/rapi/Tests/rapi08.malC
+++ b/monetdb5/extras/rapi/Tests/rapi08.malC
@@ -1,28 +1,26 @@
 gv:= bat.new(:int);
-bat.append(gv,1);
-bat.append(gv,1);
-bat.append(gv,2);
-bat.append(gv,2);
+gv:= bat.append(gv,1);
+gv:= bat.append(gv,1);
+gv:= bat.append(gv,2);
+gv:= bat.append(gv,2);
 
 dv:= bat.new(:int);
-bat.append(dv,42);
-bat.append(dv,84);
-bat.append(dv,42);
-bat.append(dv,20);
+dv:= bat.append(dv,42);
+dv:= bat.append(dv,84);
+dv:= bat.append(dv,42);
+dv:= bat.append(dv,20);
 
 # tuple identifiers
 ti:= bat.new(:oid);
-bat.append(ti,0:oid);
-bat.append(ti,1:oid);
-bat.append(ti,2:oid);
-bat.append(ti,3:oid);
+ti:= bat.append(ti,0:oid);
+ti:= bat.append(ti,1:oid);
+ti:= bat.append(ti,2:oid);
+ti:= bat.append(ti,3:oid);
 
-X_8 := algebra.projection(ti,gv);
-(X_9,r1_9,r2_9) := group.subgroupdone(X_8);
-X_12 := algebra.projection(r1_9,X_8);
-X_15 := algebra.projection(ti,dv); 
+ti_gv:bat[:int] := algebra.projection(ti,gv);
+(grp:bat[:oid],r1_9:bat[:oid],r2_9:bat[:lng]) := group.subgroupdone(ti_gv);
+ti_dv:bat[:int] := algebra.projection(ti,dv); 
 
-io.print(X_9);
-X_16:bat[:dbl] := rapi.subeval_aggr(nil:ptr,"res <- aggregate(arg2, 
by=list(aggr_group), FUN=median); return(res$x)",X_8,X_15);
-io.print(X_16);
-
+io.print(grp);
+res:bat[:dbl] := rapi.subeval_aggr(nil:ptr,"res <- aggregate(arg2, 
by=list(aggr_group), FUN=median); return(res$x)",ti_gv,ti_dv);
+io.print(res);
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

monetdb-java: default - Updated copyright year to 2017 in *.java...

2017-01-05 Thread Martin van Dinther

Changeset: b9b35ca2eec2 for monetdb-java
URL: http://dev.monetdb.org/hg/monetdb-java?cmd=changeset;node=b9b35ca2eec2
Modified Files:
build.xml
example/MJDBCTest.java
example/PreparedExample.java
example/SQLImport.java
example/SQLcopyinto.java
src/main/java/nl/cwi/monetdb/client/JMonetDB.java
src/main/java/nl/cwi/monetdb/client/JdbcClient.java
src/main/java/nl/cwi/monetdb/jdbc/MonetBlob.java
src/main/java/nl/cwi/monetdb/jdbc/MonetClob.java
src/main/java/nl/cwi/monetdb/jdbc/MonetConnection.java
src/main/java/nl/cwi/monetdb/jdbc/MonetDataSource.java
src/main/java/nl/cwi/monetdb/jdbc/MonetDatabaseMetaData.java
src/main/java/nl/cwi/monetdb/jdbc/MonetDriver.java.in
src/main/java/nl/cwi/monetdb/jdbc/MonetPreparedStatement.java
src/main/java/nl/cwi/monetdb/jdbc/MonetResultSet.java
src/main/java/nl/cwi/monetdb/jdbc/MonetSavepoint.java
src/main/java/nl/cwi/monetdb/jdbc/MonetStatement.java
src/main/java/nl/cwi/monetdb/jdbc/MonetWrapper.java
src/main/java/nl/cwi/monetdb/jdbc/types/INET.java
src/main/java/nl/cwi/monetdb/jdbc/types/URL.java
src/main/java/nl/cwi/monetdb/mcl/MCLException.java
src/main/java/nl/cwi/monetdb/mcl/io/BufferedMCLReader.java
src/main/java/nl/cwi/monetdb/mcl/io/BufferedMCLWriter.java
src/main/java/nl/cwi/monetdb/mcl/net/MapiSocket.java
src/main/java/nl/cwi/monetdb/mcl/parser/HeaderLineParser.java
src/main/java/nl/cwi/monetdb/mcl/parser/MCLParseException.java
src/main/java/nl/cwi/monetdb/mcl/parser/MCLParser.java
src/main/java/nl/cwi/monetdb/mcl/parser/StartOfHeaderParser.java
src/main/java/nl/cwi/monetdb/mcl/parser/TupleLineParser.java
src/main/java/nl/cwi/monetdb/merovingian/Control.java
src/main/java/nl/cwi/monetdb/merovingian/MerovingianException.java
src/main/java/nl/cwi/monetdb/merovingian/SabaothDB.java
src/main/java/nl/cwi/monetdb/util/CmdLineOpts.java
src/main/java/nl/cwi/monetdb/util/Exporter.java
src/main/java/nl/cwi/monetdb/util/Extract.java
src/main/java/nl/cwi/monetdb/util/OptionsException.java
src/main/java/nl/cwi/monetdb/util/SQLExporter.java
src/main/java/nl/cwi/monetdb/util/SQLRestore.java
src/main/java/nl/cwi/monetdb/util/XMLExporter.java
tests/BugConcurrent_clients_SF_1504657.java
tests/BugConcurrent_sequences.java
tests/BugDatabaseMetaData_Bug_3356.java
tests/BugDecimalRound_Bug_3561.java
tests/BugExecuteUpdate_Bug_3350.java
tests/BugSetQueryTimeout_Bug_3357.java
tests/Test_Cautocommit.java
tests/Test_Cforkbomb.java
tests/Test_CisValid.java
tests/Test_Clargequery.java
tests/Test_Cmanycon.java
tests/Test_Creplysize.java
tests/Test_Csavepoints.java
tests/Test_Csendthread.java
tests/Test_Ctransaction.java
tests/Test_Dobjects.java
tests/Test_FetchSize.java
tests/Test_Int128.java
tests/Test_PSgeneratedkeys.java
tests/Test_PSgetObject.java
tests/Test_PSlargeamount.java
tests/Test_PSlargebatchval.java
tests/Test_PSlargeresponse.java
tests/Test_PSmanycon.java
tests/Test_PSmetadata.java
tests/Test_PSsomeamount.java
tests/Test_PSsqldata.java
tests/Test_PStimedate.java
tests/Test_PStimezone.java
tests/Test_PStypes.java
tests/Test_Rbooleans.java
tests/Test_Rmetadata.java
tests/Test_Rpositioning.java
tests/Test_Rsqldata.java
tests/Test_Rtimedate.java
tests/Test_Sbatching.java
tests/Test_Smoreresults.java
tests/Test_Wrapper.java
version.sh
Branch: default
Log Message:

Updated copyright year to 2017 in *.java files and MonetDriver.java.in and 
build.xml and version.sh


diffs (truncated from 936 to 300 lines):

diff --git a/build.xml b/build.xml
--- a/build.xml
+++ b/build.xml
@@ -5,7 +5,7 @@ This Source Code Form is subject to the 
 License, v. 2.0.  If a copy of the MPL was not distributed with this
 file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
-Copyright 1997 - July 2008 CWI, August 2008 - 2016 MonetDB B.V.
+Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
 -->

monetdb-java: default - Moved content of ChangeLog to ChangeLog-...

2017-01-05 Thread Martin van Dinther

Changeset: a6608e9581c1 for monetdb-java
URL: http://dev.monetdb.org/hg/monetdb-java?cmd=changeset;node=a6608e9581c1
Modified Files:
ChangeLog
ChangeLog-Archive
release.txt
src/main/java/nl/cwi/monetdb/jdbc/MonetConnection.java
Branch: default
Log Message:

Moved content of ChangeLog to ChangeLog-Archive.
Added information on released jars to the ChangeLog-Archive, so it becomes 
clear which releases contain which fixes.
Corrected order of some entries in ChangeLog-Archive (sorted by desc date).
Updated documentation regarding createBlob() and createClob(), which are 
implemented and thus no longer throw SQLFeatureNotSupportedException.


diffs (truncated from 339 to 300 lines):

diff --git a/ChangeLog b/ChangeLog
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,82 +1,3 @@
-# ChangeLog file for java
+# ChangeLog file for monetdb-java
 # This file is updated with Maddlog
 
-* Thu Nov 17 2016 Martin van Dinther 
-- The MapiSocket object now supports getting and setting the
-  socket timeout, even before the TCP socket is created.  To set
-  the socket timeout specify it in the JDBC URL, for example:
-jdbc:monetdb://localhost:5/demo?so_timeout=8000
-  This enables the timeout and sets it to 8000 milliseconds (= 8 seconds).
-  By default the timeout is not set, so wait infinitely.
-
-* Thu Nov 10 2016 Martin van Dinther 
-- Implemented Connection methods: getClientInfo(name) and getClientInfo().
-  They used to return null and empty Properties object.
-  Corrected implementation of Connection methods: setClientInfo(name, value)
-  and setClientInfo(properties). They are now processed as expected.
-  Corrected implementation of Connection.setHoldability(holdability). It now
-  throws an SQLFeatureNotSupportedException when holdability is not
-  ResultSet.HOLD_CURSORS_OVER_COMMIT (which is the only supported holdability).
-
-* Thu Oct 13 2016 Martin van Dinther 
-- Corrected implementation of java.sql.Wrapper methods isWrapperFor()
-  and unwrap().  They now properly return expected results instead of
-  always return false or throw an SQLException.
-
-* Thu Oct  6 2016 Martin van Dinther 
-- Corrected return values of DatabaseMetaData methods nullsAreSortedHigh(),
-  nullsAreSortedLow(), getMaxCursorNameLength(), getMaxProcedureNameLength(),
-  getMaxStatementLength() and getMaxUserNameLength().
-  Improved return values of DatabaseMetaData methods 
getMaxBinaryLiteralLength(),
-  getMaxCharLiteralLength() and getMaxColumnsInTable().
-- Implemented Statement methods: getQueryTimeout() and setQueryTimeout(int
-  seconds).  getQueryTimeout() used to always return 0, now it returns the
-  query timeout retrieved from the server.  setQueryTimeout(int seconds)
-  used to always throw SQLException: query time outs not supported.
-  Now it sets the query timeout for the current connection/session on
-  the server.
-
-* Thu Sep 29 2016 Martin van Dinther 
-- Corrected ResultSet methods: getByte(), getBigDecimal(), getShort(),
-  getInt(), getLong(), getFloat() and getDouble() in case the conversion
-  to the native type failed due to a Number Format conversion error.
-  It used to silently ignore the conversion error and return 0 instead,
-  which is not correct.  Now it throws an SQLException with message
-  "Could not convert value to a number." and SQLstate "22003" meaning:
-  Numeric value out of range.
-
-* Thu Sep 22 2016 Martin van Dinther 
-- Improved JdbcClient program when doing dump of table definition.  It now
-  outputs CREATE TABLE definition more similar to mclient program.
-
-* Thu Sep 15 2016 Martin van Dinther 
-- Improved performance of following JDBC ResultSet and ResultSetMetaData
-  methods:
-  - ResultSet.getBoolean(column)
-  - ResultSet.getBinaryStream(column)
-  - ResultSet.getBytes(column)
-  - ResultSet.getObject(column)
-  - ResultSet.getObject(column, map)
-  - ResultSet.getDate(column)
-  - ResultSet.getTime(column)
-  - ResultSet.getTimestamp(column)
-  - ResultSet.getDate(column, calendar)
-  - ResultSet.getTime(column, calendar)
-  - ResultSet.getTimestamp(column, calendar)
-  - ResultSetMetaData.getColumnClassName(column)
-  - ResultSetMetaData.getColumnType(column)
-  - ResultSetMetaData.isCaseSensitive(column)
-  - ResultSetMetaData.isSigned(column)
-  - ResultSetMetaData.getPrecision(column)
-
-* Thu Sep  8 2016 Martin van Dinther 
-- Improved JdbcClient program by fixing some resource leaks.
-- Extended JdbcClient program by showing elapsed time information for each
-  query or command when started in interactive mode (no -f was used at 
startup).
-
-* Thu Sep  1 2016 Martin van Dinther

MonetDB: Dec2016 - Convert README file to restructured text

2017-01-05 Thread Panagiotis Koutsourakis

Changeset: 1804dba223dd for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1804dba223dd
Added Files:
README.rst
Removed Files:
README
Modified Files:
HowToStart.rst
Branch: Dec2016
Log Message:

Convert README file to restructured text


diffs (82 lines):

diff --git a/HowToStart.rst b/HowToStart.rst
--- a/HowToStart.rst
+++ b/HowToStart.rst
@@ -24,7 +24,7 @@ This document assumes that you are plann
 installing MonetDB on a Unix-like system (e.g., Linux, IRIX, Solaris,
 AIX, Mac OS X/Darwin, or CYGWIN).  For compilation and installation on
 a native Windows system (NT, 2000, XP) see the instructions in the
-file `../buildtools/doc/windowsbuild.rst`__.
+file `buildtools/doc/windowsbuild.rst`__.
 
 __ http://dev.monetdb.org/downloads/
 __ Windows-Installation.html
diff --git a/README b/README
deleted file mode 100644
--- a/README
+++ /dev/null
@@ -1,26 +0,0 @@
-The MonetDB Database System
-===
-
-The MonetDB database is developed by the CWI database research group
-(see http://www.monetdb.org/).
-
-Via the MonetDB project we have brought the MonetDB system in open source,
-where it is accessible at http://www.monetdb.org/Downloads/
-
-The MonetDB database system is a high-performance database kernel for
-query-intensive applications. The MonetDB kernel works together with an
-SQL frontend that is in a separate CVS module.
-
-If you got a source distribution, please compile and install MonetDB first,
-following the instructions in the file 'HowToStart.rst' (for Unix)
-or 'buildtools/doc/windowsbuild.rst' (for Windows).
-
-
-Copyright Notice
-
-
-This Source Code Form is subject to the terms of the Mozilla Public
-License, v. 2.0.  If a copy of the MPL was not distributed with this
-file, You can obtain one at http://mozilla.org/MPL/2.0/.
-
-Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
diff --git a/README.rst b/README.rst
new file mode 100644
--- /dev/null
+++ b/README.rst
@@ -0,0 +1,34 @@
+The MonetDB Database System
+===
+
+The MonetDB database is developed by the CWI database research group
+(see http://www.monetdb.org/).
+
+Via the MonetDB project we have brought the MonetDB system in open source,
+where it is accessible at http://www.monetdb.org/Downloads/
+
+The MonetDB database system is a high-performance database kernel for
+query-intensive applications. The MonetDB source can be found at our `Mercurial
+server`__. There is also a `github mirror`__ that is updated once a day.
+
+.. _MonetDB: http://dev.monetdb.org/hg/MonetDB/
+__ MonetDB_
+
+.. _github: https://github.com/MonetDB/MonetDB
+__ github_
+
+If you got a source distribution, please compile and install MonetDB first,
+following the instructions in the file `HowToStart.rst`__ (for Unix)
+or `buildtools/doc/windowsbuild.rst`__ (for Windows).
+
+__ HowToStart.rst
+__ buildtools/doc/windowsbuild.rst
+
+Copyright Notice
+
+
+This Source Code Form is subject to the terms of the Mozilla Public
+License, v. 2.0.  If a copy of the MPL was not distributed with this
+file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+Copyright 1997 - July 2008 CWI, August 2008 - 2017 MonetDB B.V.
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: default - Merge with Dec2016 branch.

2017-01-05 Thread Sjoerd Mullender

Changeset: a722eef5e949 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a722eef5e949
Modified Files:
clients/Tests/MAL-signatures.stable.out
clients/Tests/MAL-signatures.stable.out.int128
monetdb5/modules/atoms/batxml.c
monetdb5/modules/kernel/batcolor.c
monetdb5/modules/kernel/batstr.c
Branch: default
Log Message:

Merge with Dec2016 branch.


diffs (50 lines):

diff --git a/monetdb5/modules/atoms/batxml.c b/monetdb5/modules/atoms/batxml.c
--- a/monetdb5/modules/atoms/batxml.c
+++ b/monetdb5/modules/atoms/batxml.c
@@ -13,10 +13,8 @@
  * The collection of routines provided here are map operations
  * for the atom xml primitives.
  *
- * In line with the batcalc module, we assume that
- * if two bat operands are provided that they are already
- * aligned on the head. Moreover, the head of the BATs
- * are limited to :oid.
+ * In line with the batcalc module, we assume that if two bat operands
+ * are provided that they are aligned.
  *
  * The implementation is focussed on functionality. At a later stage
  * we may postpone string contstruction until it is really needed.
diff --git a/monetdb5/modules/kernel/batcolor.c 
b/monetdb5/modules/kernel/batcolor.c
--- a/monetdb5/modules/kernel/batcolor.c
+++ b/monetdb5/modules/kernel/batcolor.c
@@ -13,11 +13,8 @@
  * The collection of routines provided here are map operations
  * for the color string primitives.
  *
- * In line with the batcalc module, we assume that
- * if two bat operands are provided that they are already
- * aligned on the head. Moreover, the head of the BATs
- * are limited to :oid, which can be cheaply realized using
- * the GRPsplit operation.
+ * In line with the batcalc module, we assume that if two bat operands
+ * are provided that they are aligned.
  */
 
 #include "monetdb_config.h"
diff --git a/monetdb5/modules/kernel/batstr.c b/monetdb5/modules/kernel/batstr.c
--- a/monetdb5/modules/kernel/batstr.c
+++ b/monetdb5/modules/kernel/batstr.c
@@ -13,11 +13,8 @@
  * The collection of routines provided here are map operations
  * for the atom string primitives.
  *
- * In line with the batcalc module, we assume that
- * if two bat operands are provided that they are already
- * aligned on the head. Moreover, the head of the BATs
- * are limited to :void, which can be cheaply realized using
- * the GRPsplit operation.
+ * In line with the batcalc module, we assume that if two bat operands
+ * are provided that they are aligned.
  */
 #include "monetdb_config.h"
 #include 
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Dec2016 - Approved.

2017-01-05 Thread Sjoerd Mullender

Changeset: 74119f232ff1 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=74119f232ff1
Modified Files:
clients/Tests/MAL-signatures.stable.out
clients/Tests/MAL-signatures.stable.out.int128
Branch: Dec2016
Log Message:

Approved.


diffs (24 lines):

diff --git a/clients/Tests/MAL-signatures.stable.out 
b/clients/Tests/MAL-signatures.stable.out
--- a/clients/Tests/MAL-signatures.stable.out
+++ b/clients/Tests/MAL-signatures.stable.out
@@ -8619,8 +8619,6 @@ Ready.
 [ "url",   "url",  "command url.url(s:url):url ",  "URLnoop;", "Create 
an URL from a string literal"   ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "s2_1", "function user.s2_1():void;",   "", ""  ]
 [ "uuid",  "#cmp", "command uuid.#cmp():void ","UUIDcompare;", ""  
]
 [ "uuid",  "#fromstr", "command uuid.#fromstr():void ",
"UUIDfromString;",  ""  ]
diff --git a/clients/Tests/MAL-signatures.stable.out.int128 
b/clients/Tests/MAL-signatures.stable.out.int128
--- a/clients/Tests/MAL-signatures.stable.out.int128
+++ b/clients/Tests/MAL-signatures.stable.out.int128
@@ -10984,8 +10984,6 @@ Ready.
 [ "url",   "url",  "command url.url(s:url):url ",  "URLnoop;", "Create 
an URL from a string literal"   ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "s2_1", "function user.s2_1():void;",   "", ""  ]
 [ "uuid",  "#cmp", "command uuid.#cmp():void ","UUIDcompare;", ""  
]
 [ "uuid",  "#fromstr", "command uuid.#fromstr():void ",
"UUIDfromString;",  ""  ]
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Dec2016 - Update comments.

2017-01-05 Thread Sjoerd Mullender

Changeset: 274e1e18ebc7 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=274e1e18ebc7
Modified Files:
monetdb5/modules/atoms/batxml.c
monetdb5/modules/kernel/batcolor.c
monetdb5/modules/kernel/batstr.c
Branch: Dec2016
Log Message:

Update comments.


diffs (50 lines):

diff --git a/monetdb5/modules/atoms/batxml.c b/monetdb5/modules/atoms/batxml.c
--- a/monetdb5/modules/atoms/batxml.c
+++ b/monetdb5/modules/atoms/batxml.c
@@ -13,10 +13,8 @@
  * The collection of routines provided here are map operations
  * for the atom xml primitives.
  *
- * In line with the batcalc module, we assume that
- * if two bat operands are provided that they are already
- * aligned on the head. Moreover, the head of the BATs
- * are limited to :oid.
+ * In line with the batcalc module, we assume that if two bat operands
+ * are provided that they are aligned.
  *
  * The implementation is focussed on functionality. At a later stage
  * we may postpone string contstruction until it is really needed.
diff --git a/monetdb5/modules/kernel/batcolor.c 
b/monetdb5/modules/kernel/batcolor.c
--- a/monetdb5/modules/kernel/batcolor.c
+++ b/monetdb5/modules/kernel/batcolor.c
@@ -13,11 +13,8 @@
  * The collection of routines provided here are map operations
  * for the color string primitives.
  *
- * In line with the batcalc module, we assume that
- * if two bat operands are provided that they are already
- * aligned on the head. Moreover, the head of the BATs
- * are limited to :oid, which can be cheaply realized using
- * the GRPsplit operation.
+ * In line with the batcalc module, we assume that if two bat operands
+ * are provided that they are aligned.
  */
 
 #include "monetdb_config.h"
diff --git a/monetdb5/modules/kernel/batstr.c b/monetdb5/modules/kernel/batstr.c
--- a/monetdb5/modules/kernel/batstr.c
+++ b/monetdb5/modules/kernel/batstr.c
@@ -13,11 +13,8 @@
  * The collection of routines provided here are map operations
  * for the atom string primitives.
  *
- * In line with the batcalc module, we assume that
- * if two bat operands are provided that they are already
- * aligned on the head. Moreover, the head of the BATs
- * are limited to :void, which can be cheaply realized using
- * the GRPsplit operation.
+ * In line with the batcalc module, we assume that if two bat operands
+ * are provided that they are aligned.
  */
 #include "monetdb_config.h"
 #include 
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: default - AUTHresolveUser always allocates.

2017-01-05 Thread Sjoerd Mullender

Changeset: 51574284d8f4 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=51574284d8f4
Modified Files:
monetdb5/mal/mal_authorize.c
Branch: default
Log Message:

AUTHresolveUser always allocates.


diffs (20 lines):

diff --git a/monetdb5/mal/mal_authorize.c b/monetdb5/mal/mal_authorize.c
--- a/monetdb5/mal/mal_authorize.c
+++ b/monetdb5/mal/mal_authorize.c
@@ -91,12 +91,13 @@ AUTHrequireAdmin(Client cntxt) {
id = cntxt->user;
 
if (id != 0) {
-   char u[BUFSIZ] = "";
-   str user = u;
+   str user = NULL;
str tmp;
 
rethrow("requireAdmin", tmp, AUTHresolveUser(, id));
-   throw(INVCRED, "requireAdmin", INVCRED_ACCESS_DENIED " '%s'", 
user);
+   tmp = createException(INVCRED, "requireAdmin", 
INVCRED_ACCESS_DENIED " '%s'", user);
+   GDKfree(user);
+   return tmp;
}
 
return(MAL_SUCCEED);
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: default - Changed interfaces of the AUTH* functions.

2017-01-05 Thread Sjoerd Mullender

Changeset: 284280266e77 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=284280266e77
Modified Files:
clients/Tests/exports.stable.out
monetdb5/ChangeLog
monetdb5/mal/mal_authorize.c
monetdb5/mal/mal_authorize.h
monetdb5/mal/mal_session.c
monetdb5/modules/mal/clients.c
sql/backends/monet5/sql_user.c
tools/merovingian/daemon/controlrunner.c
tools/mserver/mserver5.c
tools/mserver/shutdowntest.c
Branch: default
Log Message:

Changed interfaces of the AUTH* functions.


diffs (truncated from 781 to 300 lines):

diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out
--- a/clients/Tests/exports.stable.out
+++ b/clients/Tests/exports.stable.out
@@ -740,19 +740,19 @@ str ALGunique1(bat *result, const bat *b
 str ALGunique2(bat *result, const bat *bid, const bat *sid);
 str ALGvariance(dbl *res, const bat *bid);
 str ALGvariancep(dbl *res, const bat *bid);
-str AUTHaddUser(oid *ret, Client c, str *user, str *pass);
-str AUTHchangePassword(Client c, str *oldpass, str *passwd);
-str AUTHchangeUsername(Client c, str *olduser, str *newuser);
-str AUTHcheckCredentials(oid *ret, Client c, str *user, str *passwd, str 
*challenge, str *algo);
-str AUTHgetPasswordHash(str *ret, Client c, str *username);
+str AUTHaddUser(oid *ret, Client c, const char *user, const char *pass);
+str AUTHchangePassword(Client c, const char *oldpass, const char *passwd);
+str AUTHchangeUsername(Client c, const char *olduser, const char *newuser);
+str AUTHcheckCredentials(oid *ret, Client c, const char *user, const char 
*passwd, const char *challenge, const char *algo);
+str AUTHgetPasswordHash(str *ret, Client c, const char *username);
 str AUTHgetUsername(str *ret, Client c);
 str AUTHgetUsers(BAT **ret1, BAT **ret2, Client c);
-str AUTHinitTables(str *passwd);
-str AUTHremoveUser(Client c, str *username);
+str AUTHinitTables(const char *passwd);
+str AUTHremoveUser(Client c, const char *username);
 void AUTHreset(void) __attribute__((__visibility__("hidden")));
-str AUTHresolveUser(str *ret, oid *uid);
-str AUTHsetPassword(Client c, str *username, str *passwd);
-str AUTHunlockVault(str *password);
+str AUTHresolveUser(str *ret, oid uid);
+str AUTHsetPassword(Client c, const char *username, const char *passwd);
+str AUTHunlockVault(const char *password);
 str BATPCREilike(bat *ret, const bat *b, const str *pat, const str *esc);
 str BATPCREilike2(bat *ret, const bat *b, const str *pat);
 str BATPCRElike(bat *ret, const bat *b, const str *pat, const str *esc);
diff --git a/monetdb5/ChangeLog b/monetdb5/ChangeLog
--- a/monetdb5/ChangeLog
+++ b/monetdb5/ChangeLog
@@ -1,6 +1,10 @@
 # ChangeLog file for MonetDB5
 # This file is updated with Maddlog
 
+* Thu Jan  5 2017 Sjoerd Mullender 
+- Changed the interfaces of the AUTH* functions: pass values, not pointers
+  to values.
+
 * Tue Dec 13 2016 Sjoerd Mullender 
 - Removed calc.setoid().
 - group.subgroup is now called group.group if it is not refining a group.
diff --git a/monetdb5/mal/mal_authorize.c b/monetdb5/mal/mal_authorize.c
--- a/monetdb5/mal/mal_authorize.c
+++ b/monetdb5/mal/mal_authorize.c
@@ -39,9 +39,9 @@
 #endif
 #endif
 
-static str AUTHdecypherValue(str *ret, str *value);
-static str AUTHcypherValue(str *ret, str *value);
-static str AUTHverifyPassword(str *passwd);
+static str AUTHdecypherValue(str *ret, const char *value);
+static str AUTHcypherValue(str *ret, const char *value);
+static str AUTHverifyPassword(const char *passwd);
 
 static BAT *user = NULL;
 static BAT *pass = NULL;
@@ -74,7 +74,7 @@ AUTHfindUser(const char *username)
if (BUNfnd(duser, ) == BUN_NONE)
return p;
}
-   } 
+   }
return BUN_NONE;
 }
 
@@ -95,7 +95,7 @@ AUTHrequireAdmin(Client cntxt) {
str user = u;
str tmp;
 
-   rethrow("requireAdmin", tmp, AUTHresolveUser(, ));
+   rethrow("requireAdmin", tmp, AUTHresolveUser(, id));
throw(INVCRED, "requireAdmin", INVCRED_ACCESS_DENIED " '%s'", 
user);
}
 
@@ -108,20 +108,21 @@ AUTHrequireAdmin(Client cntxt) {
  * InvalidCredentialsException.
  */
 static str
-AUTHrequireAdminOrUser(Client cntxt, str *username) {
+AUTHrequireAdminOrUser(Client cntxt, const char *username) {
oid id = cntxt->user;
-   char u[BUFSIZ] = "";
-   str user = u;
+   str user = NULL;
str tmp = MAL_SUCCEED;
 
/* root?  then all is well */
if (id == 0)
return(MAL_SUCCEED);
 
-   rethrow("requireAdminOrUser", tmp, AUTHresolveUser(, ));
-   if (username == NULL || *username == NULL || strcmp(*username, user) != 
0) {
+   rethrow("requireAdminOrUser", tmp, AUTHresolveUser(, id));
+   if (username == NULL || strcmp(username, user) != 0) {
+   GDKfree(user);

MonetDB: default - Approved.

2017-01-05 Thread Sjoerd Mullender

Changeset: 210598b9f526 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=210598b9f526
Modified Files:
clients/Tests/MAL-signatures.stable.out
clients/Tests/MAL-signatures.stable.out.int128
Branch: default
Log Message:

Approved.


diffs (24 lines):

diff --git a/clients/Tests/MAL-signatures.stable.out 
b/clients/Tests/MAL-signatures.stable.out
--- a/clients/Tests/MAL-signatures.stable.out
+++ b/clients/Tests/MAL-signatures.stable.out
@@ -8654,8 +8654,6 @@ Ready.
 [ "url",   "url",  "command url.url(s:url):url ",  "URLnoop;", "Create 
an URL from a string literal"   ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "s2_1", "function user.s2_1():void;",   "", ""  ]
 [ "uuid",  "#cmp", "command uuid.#cmp():void ","UUIDcompare;", ""  
]
 [ "uuid",  "#fromstr", "command uuid.#fromstr():void ",
"UUIDfromString;",  ""  ]
diff --git a/clients/Tests/MAL-signatures.stable.out.int128 
b/clients/Tests/MAL-signatures.stable.out.int128
--- a/clients/Tests/MAL-signatures.stable.out.int128
+++ b/clients/Tests/MAL-signatures.stable.out.int128
@@ -11019,8 +11019,6 @@ Ready.
 [ "url",   "url",  "command url.url(s:url):url ",  "URLnoop;", "Create 
an URL from a string literal"   ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main", "function user.main():void;",   "", ""  ]
-[ "user",  "main", "function user.main():void;",   "", ""  ]
 [ "user",  "s2_1", "function user.s2_1():void;",   "", ""  ]
 [ "uuid",  "#cmp", "command uuid.#cmp():void ","UUIDcompare;", ""  
]
 [ "uuid",  "#fromstr", "command uuid.#fromstr():void ",
"UUIDfromString;",  ""  ]
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: default - Merge with Dec2016 branch.

2017-01-05 Thread Sjoerd Mullender

Changeset: 2eb35fb7f046 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=2eb35fb7f046
Modified Files:
monetdb5/mal/mal_module.c
Branch: default
Log Message:

Merge with Dec2016 branch.

___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Dec2016 - Casting a char to an int doesn't make it posi...

2017-01-05 Thread Sjoerd Mullender

Changeset: 65e94217b6f4 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=65e94217b6f4
Modified Files:
monetdb5/mal/mal_module.c
Branch: Dec2016
Log Message:

Casting a char to an int doesn't make it positive, using unsigned char does.


diffs (48 lines):

diff --git a/monetdb5/mal/mal_module.c b/monetdb5/mal/mal_module.c
--- a/monetdb5/mal/mal_module.c
+++ b/monetdb5/mal/mal_module.c
@@ -48,13 +48,13 @@ mal_module_reset(void)
 }
 
 static void clrModuleIndex(str nme, Module cur){
-   if( moduleIndex[(int)(*nme)][(int)(*(nme+1))]== cur)
-   moduleIndex[(int)(*nme)][(int)(*(nme+1))]= cur->link;
+   if( moduleIndex[((unsigned char *) nme)[0]][((unsigned char *) 
nme)[1]]== cur)
+   moduleIndex[((unsigned char *) nme)[0]][((unsigned char 
*) nme)[1]]= cur->link;
 }
 
 static void setModuleIndex(str nme, Module cur){
-   cur->link= moduleIndex[(int)(*nme)][(int)(*(nme+1))];
-   moduleIndex[(int)(*nme)][(int)(*(nme+1))]= cur;
+   cur->link= moduleIndex[((unsigned char *) nme)[0]][((unsigned 
char *) nme)[1]];
+   moduleIndex[((unsigned char *) nme)[0]][((unsigned char *) 
nme)[1]]= cur;
 }
 
 /*
@@ -98,7 +98,7 @@ Module fixModule(Module scope, str nme){

if( strcmp(nme,"user")==0)
return scope;
-   s= moduleIndex[(int)(*nme)][(int)(*(nme+1))];
+   s= moduleIndex[((unsigned char *) nme)[0]][((unsigned char *) nme)[1]];
while(s != NULL){
if( nme == s->name )
return s;
@@ -249,7 +249,7 @@ void deleteSymbol(Module scope, Symbol p
 Module findModule(Module scope, str name){
Module def=scope;
if( name==NULL) return scope;
-   scope= moduleIndex[(int)(*name)][(int)(*(name+1))];
+   scope= moduleIndex[((unsigned char *) name)[0]][((unsigned char *) 
name)[1]];
while(scope != NULL){
if( name == scope->name )
return scope;
@@ -262,7 +262,7 @@ Module findModule(Module scope, str name
 int isModuleDefined(Module scope, str name){
if( name==NULL || scope==NULL) return FALSE;
if( name == scope->name) return TRUE;
-   scope= moduleIndex[(int)(*name)][(int)(*(name+1))];
+   scope= moduleIndex[((unsigned char *) name)[0]][((unsigned char *) 
name)[1]];
while(scope != NULL){
if( name == scope->name )
return TRUE;
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Dec2016 - Unnecessary tests.

2017-01-05 Thread Sjoerd Mullender

Changeset: 1e62b63b5b74 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1e62b63b5b74
Modified Files:
monetdb5/mal/mal_module.c
Branch: Dec2016
Log Message:

Unnecessary tests.


diffs (23 lines):

diff --git a/monetdb5/mal/mal_module.c b/monetdb5/mal/mal_module.c
--- a/monetdb5/mal/mal_module.c
+++ b/monetdb5/mal/mal_module.c
@@ -84,8 +84,7 @@ Module newModule(Module scope, str nme){
// User modules are never global
if( strcmp(nme,"user")){
setModuleIndex(nme,cur);
-   if ( moduleChain)
-   cur->next = moduleChain;
+   cur->next = moduleChain;
moduleChain = cur;
}
return cur;
@@ -99,8 +98,7 @@ Module fixModule(Module scope, str nme){

if( strcmp(nme,"user")==0)
return scope;
-   if( moduleIndex[(int)(*nme)][(int)(*(nme+1))])
-   s= moduleIndex[(int)(*nme)][(int)(*(nme+1))];
+   s= moduleIndex[(int)(*nme)][(int)(*(nme+1))];
while(s != NULL){
if( nme == s->name )
return s;
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: Dec2016 - Remove module from chain when freeing.

2017-01-05 Thread Sjoerd Mullender

Changeset: 2e0771f224f4 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=2e0771f224f4
Modified Files:
monetdb5/mal/mal_module.c
Branch: Dec2016
Log Message:

Remove module from chain when freeing.


diffs (52 lines):

diff --git a/monetdb5/mal/mal_module.c b/monetdb5/mal/mal_module.c
--- a/monetdb5/mal/mal_module.c
+++ b/monetdb5/mal/mal_module.c
@@ -27,8 +27,8 @@
  *
  * All modules are persistent during a server session
  */
-Module moduleIndex[256][256];  /* to speedup access to correct scope */
-Module moduleChain;/* keep the modules in a chain 
as well */
+Module moduleIndex[256][256];  /* to speedup access to correct scope */
+static Module moduleChain; /* keep the modules in a chain as well 
*/
 
 static void newModuleSpace(Module scope){
scope->space = (Symbol *) GDKzalloc(MAXSCOPE * sizeof(Symbol));
@@ -42,14 +42,9 @@ getModuleChain(void){
 void
 mal_module_reset(void)
 {
-   Module m,n;
-
-   for( m = moduleChain, moduleChain = 0; m; ){
-   n = m->next;
-   freeModule(m);
-   m= n;
-   }
-   memset(moduleIndex, 0, 256 * 256 * sizeof(Module));
+   while (moduleChain)
+   freeModule(moduleChain);
+   memset(moduleIndex, 0, sizeof(moduleIndex));
 }
 
 static void clrModuleIndex(str nme, Module cur){
@@ -149,6 +144,19 @@ void freeModule(Module m)
(void)ret;
}
}
+   if (moduleChain == m)
+   moduleChain = m->next;
+   else {
+   Module tm = moduleChain;
+   while (tm && tm->next) {
+   if (tm->next == m) {
+   tm->next = m->next;
+   break;
+   }
+   tm = tm->next;
+   }
+   }
+   m->next = NULL;
freeSubScope(m);
clrModuleIndex(m->name, m);
if (m->help)
___
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list

MonetDB: wlcr - mereg with default

MonetDB: wlcr - Create the replica server process

MonetDB: wlcr - fix compilation

MonetDB: wlcr - Enable startup testing

MonetDB: wlcr - Only call wlcr optimizer when needed.

MonetDB: wlcr - Prepare for reading single MAL statements from s...

MonetDB: wlcr - Assertion off by one test solved

MonetDB: wlcr - Merge with default

MonetDB: wlcr - Save the state before patching MAL parser

MonetDB: wlcr - Basic handling of WLCR logs writes

MonetDB: wlcr - merge with default

MonetDB: wlcr - WorkLoad Capture and Replay

monetdb-java: embedded - Made fixings regarding the null values ...

MonetDB: Dec2016 - Adding known referential integrity check test...

MonetDB: default - Merge with Dec2016 branch.

MonetDB: Dec2016 - Don't shift with a negative amount or with to...

MonetDB: Dec2016 - No unaligned memory access.

MonetDB: Dec2016 - README was converted to restructured text.

MonetDB: Dec2016 - Backported test from default branch.

monetdb-java: default - Updated copyright year to 2017 in *.java...

monetdb-java: default - Moved content of ChangeLog to ChangeLog-...

MonetDB: Dec2016 - Convert README file to restructured text

MonetDB: default - Merge with Dec2016 branch.

MonetDB: Dec2016 - Approved.

MonetDB: Dec2016 - Update comments.

MonetDB: default - AUTHresolveUser always allocates.

MonetDB: default - Changed interfaces of the AUTH* functions.

MonetDB: default - Approved.

MonetDB: default - Merge with Dec2016 branch.

MonetDB: Dec2016 - Casting a char to an int doesn't make it posi...

MonetDB: Dec2016 - Unnecessary tests.

MonetDB: Dec2016 - Remove module from chain when freeing.

32 matches

Site Navigation

Mail list logo

Footer information