Bugs item #1766259, was opened at 2007-08-02 17:20
Message generated for change (Comment added) made by sjoerd
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1766259&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: Pathfinder CVS Head
>Status: Closed
Resolution: Fixed
Priority: 6
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Sjoerd Mullender (sjoerd)
Summary: XQ: monetdb immune for updates
Initial Comment:
this is another try at reproducing errors i get. (using monday's nightly build
containing sjoerd's fixes for first swizzle bug)
given a set of documents (they are again too large to be included as attachment
at sourceforge, and i've sent them to sjoerd again) and executing a list of
updates (by typing 'MapiClient -lxq < scenario.xq') will result in a (large)
document. it seems to be broken though, because when doing another update:
echo 'do insert <aap/> into doc("test003.xml")/id("17990")' | MapiClient
the document is not updated (no 'aap' is inserted) :
echo 'doc("test003.xml")/id("17990")' | MapiClient
<file xend="36325078" xstart="36324973"
name="MANIFEST.MF"><size>106</size><modified><date
unixtime="1169823760"/></modified><compressedsize>94</compressedsize><container
tool="general-purpose/magic-tool"><magic>text/plain;
charset=us-ascii</magic></container></file>
----------------------------------------------------------------------
>Comment By: Sjoerd Mullender (sjoerd)
Date: 2008-02-11 16:00
Message:
Logged In: YES
user_id=43607
Originator: NO
This error had nothing to do with the immune for updates problem. It
occurred in the call to pf:collections() at the start of the test.
The problem was in the unoptimized multiplex code where the head column
was copied to the result BAT. This was a GDK-2 bug.
No test needed.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-02-10 20:40
Message:
Logged In: YES
user_id=572415
Originator: NO
File Added: SF-1766259.debug.output
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-02-10 20:39
Message:
Logged In: YES
user_id=572415
Originator: NO
When run in isolation, the test works fine.
Only when run after the preceding tests in the same directory the test
fails as reported below.
I have not (yet?) analyzed, which (combination of) preceding tests makes
this test(s) fail.
However, the attached "debugging" patch for pathfinder.mx and the
respective output suggest that
(1) the call to xmltab4() in ws_collections() is the failing one, and
(2) the failure is triggered by the head of coll_upd being materialized by
a "[+](BAT[VoID,str],BAT[VoID,str])" received to inputs of different length
because the coll_* are (unexpectedly? / incorrectly?) not aligned.
Here, it's time for some XQuery updated expert(s) to join in / take over
...
File Added: SF-1766259.debug.patch
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-01-22 15:04
Message:
Logged In: YES
user_id=572415
Originator: NO
re-opened as the test still (again) fails, though now differently than
originally reported:
ERROR = !ERROR: interpret: no matching MIL operator to
'xmltab4(BAT[oid,BAT], str, BAT[oid,str], str, BAT[oid,str], str,
BAT[oid,str], str, BAT[oid,str])'.
!MAYBE YOU MEAN:
! xmltab4(BAT[void,BAT], str, BAT[void,str], str, BAT[void,str],
str, BAT[void,str], str, BAT[void,str]) : BAT[void,oid]
!ERROR: interpret_params: cross(param 2): evaluation error.
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugTracker/immune_for_updates.SF-1766259.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Current/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugTracker/immune_for_updates.SF-1766259.err.00.html
----------------------------------------------------------------------
Comment By: Sjoerd Mullender (sjoerd)
Date: 2007-12-07 15:29
Message:
Logged In: YES
user_id=43607
Originator: NO
The bug has been fixed.
Stefan rightly located one problem, and his tentative fix was correct, but
not sufficient.
There was another place where there was a confusion between RID and PRE
numbers which in this case mattered, since a new page had been inserted
between the original pages 0 and 1.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2007-10-07 22:55
Message:
Logged In: YES
user_id=572415
Originator: NO
Intensive debugging revealed that the "CMDfind: value not found." error
occurs in pf_support.mx's
PROC movedata(bat[void,bat] ws, oid cont, oid from, int size, int delta):
void
in line 2379:
2375: if (isoldpage) {
2376: if (rid_nid_update.exist(from_rid)) {
2377: from_nid := rid_nid_update.find(from_rid);
2378: } else {
2379: from_nid := pre_nid.find(swizzle(from_rid, map_pid_update));
2380: }
2381: } else {
2382: from_nid := rid_nid.find(from_rid);
2383: }
As far as I can tell, it looks as if the code tries to access a new page
in pre_nid that has not been added, yet; here are some details in cases
where the error occurs:
from = [EMAIL PROTECTED]
from_rid = [EMAIL PROTECTED]
swizzle(from_rid, map_pid_update) = [EMAIL PROTECTED]
pageno = [EMAIL PROTECTED]
pageid = [EMAIL PROTECTED]
isoldpage = true
|rid_nid_update| = 16289
|map_pid_update| = 3
|map_pid| = 2
|rid_nid| = 49152
|pre_nid| = 32768
Obviously, swizzle(from_rid, map_pid_update)[EMAIL PROTECTED] does not occur in
the dense head of pre_nid.
My initial idea of changing
2379: from_nid := pre_nid.find(swizzle(from_rid, map_pid_update));
into
2379: from_nid := pre_nid.find(swizzle(from_rid, map_pid));
did not help. It made the error message vanish, but some of the inserts
got lost without trace.
Splitting the original failing query
let $todo := subsequence(doc("prepared49736.data")/XIRAF/item[*], 1537,
256)
...
into two consecutive complementary queries
let $todo := subsequence(doc("prepared49736.data")/XIRAF/item[*], 1537,
$a)
...
<>
let $todo := subsequence(doc("prepared49736.data")/XIRAF/item[*], 1537+$a,
256-$a)
...
revealed that the reported bug only occurs with 0 <= $a <= 68 (i.e., 256
>= 256-$a >= 188) and 195 <= $a <= 256 (i.e., 61 >= 256-$a >= 0).
With that, I reached the limit of my humble update implementation
expertise, and hence feel free to hand it over to the real update experts
Peter & Sjoerd.
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2007-10-07 16:27
Message:
Logged In: YES
user_id=621590
Originator: YES
i meant that the error:
QUERY = let $todo := subsequence(doc("prepared49736.data")/XIRAF/item[*],
1537,256)
let $dest := for $i in doc("test10.xml")//[EMAIL PROTECTED]
where some $k in $todo/@xid satisfies $k = $i/@xid
return $i
for $item in $todo
return
do insert $item/*
into exactly-one([EMAIL PROTECTED] = $item/@xid]])
>
let $todo := subsequence(doc("prepared49736.data")/XIRAF/item[*],
1537,256)
let $dest := for $i in doc("test10.xml")//[EMAIL PROTECTED]
where some $k in $todo/@xid satisfies $k = $i/@xid
ERROR = !parse error: syntax error, unexpected let $ on line 12, column 6
(next token is `let $')
has gone. the "CMDfind: value not found." error still occurs, as you can
see in the testweb output :)
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2007-10-06 16:51
Message:
Logged In: YES
user_id=572415
Originator: NO
Thanks.
What exactly do you mean with "the parser error is gone"?
Does no error at all occur any more, or is it now back to your originally
reported "CMDfind: value not found." error?
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2007-10-06 16:07
Message:
Logged In: YES
user_id=621590
Originator: YES
I added a node-counting query to the BugTracker-test to determine whether
the updates succeeded.
I also added (expected) stable output.
Somehow by adding the following lines to the xq-script, the parser error
is gone:
"
<>
count(doc("test10.xml")//*)
"
should this be reported in a separate bug-report?
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2007-10-05 00:01
Message:
Logged In: YES
user_id=572415
Originator: NO
Wouter,
can you check the output of the first n-1 queries in your test (those that
do not trigger an error) and approve the output for all queries that yield
correct results/output?
Simply run the test, check all output and fix incorrect output by hand, if
you know the correct one,
or otherwise replace it by, say,
! Correct/expected output of query N still needs to be provided/approved
!
That would help us very much with debugging!
Thanks in advance!
Stefan
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2007-10-03 09:57
Message:
Logged In: YES
user_id=621590
Originator: YES
that was the initial error i also got, but i thought i did something
wrong, because when i did the following afterwards:
- stop monetdb
- delete the dbfarm (and log-folder)
- start monetdb
- rerun the test-script
it gave me the error as i reported it earlier: ERROR = !ERROR: CMDfind:
value not found.
could it be a issue with mapiclient's blocksize?
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2007-10-03 08:47
Message:
Logged In: YES
user_id=572415
Originator: NO
see also
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugTracker/immune_for_updates.SF-1766259.out.00.html
http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d-Fedora6/tests_BugTracker/immune_for_updates.SF-1766259.err.00.html
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2007-10-02 22:42
Message:
Logged In: YES
user_id=572415
Originator: NO
Attached the output (stderr) I get.
File Added: immune_for_updates.SF-1766259.test.err
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2007-10-02 22:41
Message:
Logged In: YES
user_id=572415
Originator: NO
Attached the output (stdout) I get.
File Added: immune_for_updates.SF-1766259.test.out
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2007-10-02 22:40
Message:
Logged In: YES
user_id=572415
Originator: NO
Waiting for 4.5 minutes (optimized MonetDB/XQuery 0.20 build on 2GHz
Dual-Core Athlon 64), I get quite some output (no idea, whether that's
correct or not) and then (as first and only error):
QUERY = let $todo := subsequence(doc("prepared49736.data")/XIRAF/item[*],
1537,256)
let $dest := for $i in doc("test10.xml")//[EMAIL PROTECTED]
where some $k in $todo/@xid satisfies $k = $i/@xid
return $i
for $item in $todo
return
do insert $item/*
into exactly-one([EMAIL PROTECTED] = $item/@xid]])
>
let $todo := subsequence(doc("prepared49736.data")/XIRAF/item[*],
1537,256)
let $dest := for $i in doc("test10.xml")//[EMAIL PROTECTED]
where some $k in $todo/@xid satisfies $k = $i/@xid
ERROR = !parse error: syntax error, unexpected let $ on line 12, column 6
(next token is `let $')
!parse error: XQuery parsing failed
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2007-10-02 21:31
Message:
Logged In: YES
user_id=621590
Originator: YES
i filed a test under
tests/BugTracker/Tests/immune_for_updates.SF-1766259.*
i did not add stable output. it currently returns:
ERROR = !ERROR: CMDfind: value not found.
!ERROR: CMDfind: operation failed.
on the last query.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2007-09-29 14:07
Message:
Logged In: YES
user_id=572415
Originator: NO
Wouter,
could you add your latest test to CVS?
Thanks in advance!
Stefan
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2007-09-25 12:01
Message:
Logged In: YES
user_id=621590
Originator: YES
next try... (sorry for the delay)
i could not find a smaller script with the exact same details of this bug,
but this one comes close (and the result is again that the document seems
to be broken and no updates can be performed afterwards).
i attached a script (+xml files) that should run within a minute with
Monetdb claiming less than 500MB of memory. It produces the error message:
ERROR = !ERROR: CMDfind: value not found.
!ERROR: CMDfind: operation failed.
on the last query of the script, which tries to insert 256 elements (each
with 2 child elements (total=768 element-nodes)) into the test10.xml
document.
trying to insert those elements in batches of 100, 100 and 56 elements
(instead of 256 at once) seems to work, but afterwards the document is
corrupted. (for example: adding the first batch of 100 again, yields only
101 new element-nodes (instead of the expected 300))
(perhaps this helps to find the cause: i noticed that it is still possible
to insert elements (say: '<aap/>') into nodes that occur in the beginning
of the document (for example the node with xid=10))
p.s. i used the 0.18.4 Superball, compiled with 64bit, 64bit-oids,
optimization enabled, on fedora core 6
File Added: test10.tar.gz
----------------------------------------------------------------------
Comment By: Martin Kersten (mlkersten)
Date: 2007-09-09 11:00
Message:
Logged In: YES
user_id=490798
Originator: NO
Wouter, please check status and update/close the report.
----------------------------------------------------------------------
Comment By: Sjoerd Mullender (sjoerd)
Date: 2007-08-03 16:13
Message:
Logged In: YES
user_id=43607
Originator: NO
I've tried running your test, but after nearly two hours of thrashing I
killed the Mserver so I could use my machine again.
Please, if you think this is a bug and you want to get it fixed, come up
with a smaller test.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1766259&group_id=56967
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
