Bugs item #2417515, was opened at 2008-12-11 17:04 Message generated for change (Settings changed) made by boncz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2417515&group_id=56967
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core Group: MonetDB4 "stable" >Status: Closed >Resolution: Invalid Priority: 5 Private: No Submitted By: Hans (hans_0_) Assigned to: Sjoerd Mullender (sjoerd) Summary: MonetDB database corrupt on restart shortly after an update Initial Comment: Hi MonetDB development team, The MonetDB database gets corrupt when Mserver is killed shortly after a series of xqueries containing add-doc() and insert statements. Starting Mserver shows problems while loading the pathfinder module. I downloaded Monet Fedora RPMs from: http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./20081209/RPMs/Fedora9.i386.oid32/ I installed these with yum on my Fedora fc9 installation, running within VMWare on my Windows XP 32 bit machine. On request, a copy of the VMware virtual machine is available for analysis. I performed the 4 steps as described in the text following this intro. For step 2 I use a couple of Java programs that interact with Mserver via a JDBC connection. The xquery log of these Java programs is included. After this I also cannot gain access to Mserver with mclient: mclient -p 51240 -l xq !xchange_challenge: frontend xquery not found Regards, Hans van Rijswijk =================================================================================================================== =================================================================================================================== ============== Step 1: boot Mserver =================================================================================================================== =================================================================================================================== Command: ($MONETDB_HOME/bin/Mserver --config=$MONETDB_CONFIG_FILE --dbinit="module(pathfinder);mil_start();" --set xquery_backend=algebra)& =================================================================================================================== =================================================================================================================== Config script: # Configuration file for MonetDB main-memory database xiraf_pid_file=/var/xiraf/recover_monetdb_frontend.PID xiraf_current_mserver_pid_file=/var/xiraf/current_mserver_frontend.PID xiraf_recover_log=/var/xiraf/log/recover_frontend.log xiraf_monetdb_lives=9 xiraf_transfer_folder=/var/xiraf/dbtransfer # This startup file is read by the GDK kernel, to set # system wide parameters. It can reside in either the local directory # where Mserver is called, or the home directory of the MonetDB user. # Where MonetDB is installed prefix=/usr exec_prefix=/usr datadir=/usr/share # GDK Section #============ # All databases are stored as a directory under the directory # given below. Use this value to change the overall location # of your data. gdk_dbfarm=/var/xiraf/MonetDB4/dbfarm-frontend # MonetDB can only handle one database at a time. You can specify # the name of the database to use below. Note that you can use # multiple databases, but that you have to edit the database name # and restart MonetDB. gdk_dbname=demo # The options below deal with various aspects of memory usage behaviour. # If you don't know what these values mean, you should leave them alone. # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously #gdk_mem_bigsize=262144 # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped; #gdk_vm_minsize=137438953472 # for 32 bit systems try to claim 3G of (virtual) memory; #gdk_vmtrim=yes # set to "no" to disable the vmtrim thread. # valgrind issues: # for 32 bit systems it tries claim 3G of (virtual) memory # set to "no" to run Mserver with valgrind. # also, you need to tell valgrind to use 8-byte alignment, hence: # "valgrind --alignment=8 Mserver --set gdk_alloc_map=no ..." gdk_alloc_map=no # the kernel can be used in embedded situations (don't set this yourself) gdk_embedded=no # You can enable debug for specific parts. By default debug is # switched off. gdk_debug=0 # possible values: (can be a combination of) # 1 = thread-specific debug output # 2 = property enforcing on new BATs # 4 = memory allocation # 8 = property checking on all values: tells about wrongly set properties # 16 = major IO activity # 32 = BAT handling # 64 = MIL parser debugging # 128 = Thread management # 256 = Event management # 512 = Transaction management # 1024 = Locks and Triggers # 2048 = Dynamic loading # 4096 = BBP Performance (?) # 8192 = Delta debugging (?) # 16384 = Module loading # 32768 = Yacc specific error messages # 65536 = tcpip (??) # 131072 = monet_multiplex: setaggr debugging (??) # 262144 = \ # 524288 = ddbench (???) # 1048576 = / # 2097152 = show join/select algorithm chosen # 4194304 = show result size estimations (for join,select) # 8388608 = extended property checking: tells also about not set properties #16777216 = disable property checking with join & outerjoin (e.g., for performance measurements) #33554432 = disable "cleaning" of freed memory in GDKfree() (e.g., for performance measurements) # General client Section #======================= #host=localhost #the host the server runs on # MonetDB Section #================ # Administrator user name monet_admin=adm # Prompt when using MIL interface monet_prompt=MonetDB> # Show version and copyright information when starting the server or # a MIL client? monet_welcome=yes # Where do the MonetDB modules reside? # The first directory contains MIL modules, and either the second or third (depending on the system) the C modules. monet_mod_path=/usr/lib/MonetDB4:/usr/lib/MonetDB4/lib:/usr/lib/MonetDB4/bin # Start Mserver in daemon mode, or start with MIL interactive interface? monet_daemon=yes # Monet Application Interface Section #==================================== # default port to address an Mserver for a MIL session mapi_port=51240 #allow mclient sessions from outside the server machine mapi_open=true # show all the MIL output? (default: no) #mapi_noheaders=no # The MAPI debug level (should be disabled for normal use) mapi_debug=0 # The number of concurrent clients can be set using mapi_clients # (smaller than 1 is not very useful) #mapi_clients=2 # SQL Interface Section #====================== # The SQL debug level (should be disabled for normal use) sql_debug=0 # Place to store transaction logs # The logs are typically stored on a different storage medium # to protect the database against accidental hardware loss. # it should be different directory as used for xquery! sql_logdir=/var/MonetDB4/sql_logs # XQuery Interface Section #========================= #default back-end: "milprint_summer" or "algebra" #xquery_backend=algebra #output mode "dm" = pretty printed, # "xml" = enforce XML with enclosing <XQueryResult> tags xquery_output=dm #limit the document cache on disk to 100MB xquery_cacheMB=100 #rules like {URL=seconds;}* e.g. cache w3c for a day, CNN for 5 minutes #xquery_cacherules="http://www.w3.org/=79200;http://www.cnn.com=300"; xquery_cacherules="" #limit the function cache in memory to 128MB xquery_procMB=128 # Place to store transaction logs # The logs are typically stored on a different storage medium # to protect the database against accidental hardware loss. # it should be different directory as used for sql! xquery_logdir=/opt/xiraf/MonetDB/var/MonetDB4/xquery_logs_frontend # if not set port to address the HTTP server for pathfinder = mapi_port+1 xrpc_port="" # xrpc daemon is open for outside connections by default xrpc_open=false # Additional options to be passed to the XRPC server, currently only # "timing". xrpc_options="" # IP addresses or hostname separated by ';' who can execute scripts in # <datadir>/MonetDB/xrpc/admin/ xrpc_admin="http://127.0.0.1;127.0.0.1;http://localhost;localhost"; # IP addresses or hostname separated by ';' who can retrieve XML # documents stored in the database using a URL of the form: # http://<host>[:port]/xrpc/<name>.xml xrpc_user="http://127.0.0.1;127.0.0.1;http://localhost;localhost"; # url prefixes of trusted modules separated by ';' xrpc_trusted="http://monetdb.cwi.nl;http://127.0.0.1;http://localhost;/usr/share/MonetDB/xrpc/export"; # standoff annotation XPath extension #standoff_ns=http://monetdb.cwi.nl/standoff standoff_ns= standoff_start=start standoff_end=end # Profile Interface Section #==================================== # for 32 bit systems try to claim 3G of (virtual) memory; # set to "no" to run Mserver with valgrind. # valgrind --tool=callgrind Mserver --set gdk_alloc_map=no ..." #gdk_alloc_map=yes =================================================================================================================== =================================================================================================================== ============== Step 2: Insert data via JDBC connection =================================================================================================================== =================================================================================================================== Query log: Thu Dec 11 13:42:55 CET 2008 executeQuery: element result {let $projdoc := pf:documents()[.='projects.xml'] return if (exists($projdoc)) then element projectsdocexists {} else element projectsdocnonexistent {}} Thu Dec 11 13:42:55 CET 2008 insertXMLDoc: pf:add-doc("/var/xiraf/dbtransfer/xiraf4040814192720964387.xml","projects.xml","projects.xml_1228999375401",10) Thu Dec 11 13:42:55 CET 2008 executeQuery: element result {let $projdoc := pf:documents()[.='bookmarks.xml'] return if (exists($projdoc)) then element docexists {} else element docnonexistent {}} Thu Dec 11 13:42:55 CET 2008 insertXMLDoc: pf:add-doc("/var/xiraf/dbtransfer/xiraf3326179746776330670.xml","bookmarks.xml","bookmarks.xml_1228999375870",10) Thu Dec 11 13:42:55 CET 2008 executeQuery: element result {let $projdoc := pf:documents()[.='reports.xml'] return if (exists($projdoc)) then element docexists {} else element docnonexistent {}} Thu Dec 11 13:42:56 CET 2008 insertXMLDoc: pf:add-doc("/var/xiraf/dbtransfer/xiraf6349201089799532961.xml","reports.xml","reports.xml_1228999376208",10) Thu Dec 11 13:43:09 CET 2008 executeQuery: doc("projects.xml")/projects/proje...@name='demo'] Thu Dec 11 13:43:09 CET 2008 executeQuery: 1+1 Thu Dec 11 13:43:09 CET 2008 hasXMLDoc: count(pf:documents()[.="demo.xml"]) Thu Dec 11 13:43:09 CET 2008 hasXMLDoc: count(pf:documents()[.="notes_demo.xml"]) Thu Dec 11 13:43:09 CET 2008 hasXMLDoc: count(pf:documents()[.="demo.xml"]) Thu Dec 11 13:43:10 CET 2008 insertXMLDoc: pf:add-doc("/var/xiraf/dbtransfer/xiraf493570239516592169.xml","demo.xml","demo.xml_1228999390017") Thu Dec 11 13:43:10 CET 2008 hasXMLDoc: count(pf:documents()[.="notes_demo.xml"]) Thu Dec 11 13:43:10 CET 2008 insertXMLDoc: pf:add-doc("/var/xiraf/dbtransfer/xiraf7574288057964248812.xml","notes_demo.xml","notes_demo.xml_1228999390164",10) Thu Dec 11 13:43:10 CET 2008 executeQuery: element result {let $projdoc := pf:documents()[.='projects.xml'] return if (exists($projdoc)) then ( element projectsdocexists {} , let $created := doc("projects.xml")/projects/proje...@name='demo'] return if (exists($created)) then element projectcreated {} else element projectnotcreated {} ) else element projectsdocnonexistent {},let $proj := pf:documents()[.='demo.xml'] return if (exists($proj)) then element projectxmlexists {} else element projectxmlnonexistent {}} Thu Dec 11 13:43:10 CET 2008 executeUpdate: let $proj := element project { attribute name {"demo"}, element index {"demo.xml"}, element casename {"demo"}, element notes {"notes_demo.xml"}, element anonymous {"nesag8"}, element description {""}} return do insert $proj as last into doc("projects.xml")/projects Thu Dec 11 13:43:10 CET 2008 hasXMLDoc: count(pf:documents()[.="restored-document.xml"]) Thu Dec 11 13:43:10 CET 2008 insertXMLDoc: pf:add-doc("/var/xiraf/dbtransfer/xiraf1949015156574180619.xml","restored-document.xml","restored-document.xml_1228999390760") Thu Dec 11 13:43:10 CET 2008 executeQuery: for $f in doc("restored-document.xml")/bookmark/folder return (string($f/@name), pf:nid(exactly-one($f/bookmark-definition))) Thu Dec 11 13:43:10 CET 2008 deleteXMLDoc: pf:del-doc("restored-document.xml") Thu Dec 11 13:43:10 CET 2008 hasXMLDoc: count(pf:documents()[.="restored-document.xml"]) Thu Dec 11 13:43:11 CET 2008 insertXMLDoc: pf:add-doc("/var/xiraf/dbtransfer/xiraf2115018461422569140.xml","restored-document.xml","restored-document.xml_1228999391045") Thu Dec 11 13:43:11 CET 2008 executeQuery: for $f in doc("restored-document.xml")/report/folder return (string($f/@name), pf:nid(exactly-one($f/report-definition))) Thu Dec 11 13:43:11 CET 2008 deleteXMLDoc: pf:del-doc("restored-document.xml") =================================================================================================================== =================================================================================================================== ============== Step 3: shutdown Mserver =================================================================================================================== =================================================================================================================== Command: kill -TERM $MONETDB_PID =================================================================================================================== =================================================================================================================== ============== Step 4: boot Mserver =================================================================================================================== =================================================================================================================== Command: ($MONETDB_HOME/bin/Mserver --config=$MONETDB_CONFIG_FILE --dbinit="mil_start();module(pathfinder);" --set xquery_backend=algebra)& =================================================================================================================== =================================================================================================================== Errors: # MonetDB Server v4.26.1 # based on GDK v1.26.1 # Copyright (c) 1993-July 2008, CWI. All rights reserved. # Copyright (c) August 2008-, MonetDB B.V.. All rights reserved. # Compiled for i686-redhat-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/pftijah !ERROR: logger: could not use bat (220) for 1000000005_map_pid !ERROR: logger: could not use bat (780) for 1000000005_rid_size !ERROR: logger: could not use bat (781) for 1000000005_rid_level !ERROR: logger: could not use bat (735) for 1000000005_rid_prop !ERROR: logger: could not use bat (782) for 1000000005_rid_kind !ERROR: logger: could not use bat (242) for 1000000005_rid_nid !ERROR: logger: could not use bat (748) for 1000000005_nid_rid !ERROR: logger: could not use bat (750) for 1000000005_qn_histogram !ERROR: logger: could not use bat (447) for 1000000005_qn_prefix_uri_loc !ERROR: logger: could not use bat (228) for 1000000005_qn_uri_loc !ERROR: logger: could not use bat (515) for 1000000005_qn_prefix !ERROR: logger: could not use bat (343) for 1000000005_qn_uri !ERROR: logger: could not use bat (326) for 1000000005_qn_loc !ERROR: logger: could not use bat (757) for 1000000005_attr_own !ERROR: logger: could not use bat (786) for 1000000005_attr_qn !ERROR: logger: could not use bat (785) for 1000000005_attr_prop !ERROR: logger: could not use bat (244) for 1000000006_map_pid !ERROR: logger: could not use bat (761) for 1000000006_rid_size !ERROR: logger: could not use bat (755) for 1000000006_rid_level !ERROR: logger: could not use bat (702) for 1000000006_rid_prop !ERROR: logger: could not use bat (706) for 1000000006_rid_kind !ERROR: logger: could not use bat (230) for 1000000006_nid_rid !ERROR: logger: could not use bat (525) for 1000000006_qn_uri_loc !ERROR: logger: could not use bat (564) for 1000000006_qn_prefix !ERROR: logger: could not use bat (523) for 1000000006_qn_uri !ERROR: logger: could not use bat (516) for 1000000006_qn_loc !ERROR: logger: could not use bat (736) for 1000000006_prop_text !ERROR: logger: could not use bat (372) for 1000000006_prop_com !ERROR: logger: could not use bat (783) for 1000000006_prop_ins !ERROR: logger: could not use bat (316) for 1000000006_prop_tgt !ERROR: logger: could not use bat (347) for 1000000006_prop_val !ERROR: logger: could not use bat (749) for 1000000006_attr_own !ERROR: logger: could not use bat (747) for 1000000006_attr_qn =================================================================================================================== =================================================================================================================== ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2008-12-12 10:33 Message: I found another way of reproducing the errors: by doing a sync() before the server is killed. >From your mailing list posting I remembered that you used sync() at some point. Don't use sync() or commit(). They just don't work in combination with the write-ahead log. ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2008-12-12 10:14 Message: The only way I can reproduce this scenario is by only partially cleaning up the database. The database consists of two important parts: the dbfarm directory and the xquery_logs directory. If I delete (the contents of) the dbfarm directory, but not (the contents of) the xquery_logs directory, I get these same error messages when I restart the server afterwards. Note the definitions of these variables in your config file: gdk_dbfarm=/var/xiraf/MonetDB4/dbfarm-frontend xquery_logdir=/opt/xiraf/MonetDB/var/MonetDB4/xquery_logs_frontend It is these two directories that should be cleared together when removing the database. ---------------------------------------------------------------------- Comment By: Peter Boncz (boncz) Date: 2008-12-11 23:49 Message: Sjoerd: it looks like bats are non-existent after the server is restarted. I think it may have something to do with the logger changes you carried through (the "leftover" bugs). I am not sure now when a subcommit is forced. The subcommit should checkpoint the new bats and write BBP.dir. After that, the bats should exist. Is a subcommit forced after each document mgt query? I guess it must be, because afaik we do not implement logging a shred action. In any case, the log gives a good handle for a replay, and the VM image will be handy too. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2417515&group_id=56967 ------------------------------------------------------------------------------ _______________________________________________ Monetdb-bugs mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
