Bugs item #1991738, was opened at 2008-06-12 11:30
Message generated for change (Comment added) made by vzzzbx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1991738&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "stable"
>Status: Closed
Resolution: Fixed
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Wouter Alink (vzzzbx)
Summary: XQ: compilation or runtime choice
Initial Comment:
(stable nightly build, compiled from supersourcetarball, June 11, 2008)
The following query takes about 35 minutes to compile (and I have reason to
believe it was also the reason why the machine crashed after it was executed a
few times):
for $run in pf:collection("log.xml")//run
let $successCount := if ($run/success) then count($run/success/task) else 0
let $failureCount := if ($run/failure) then count($run/failure/task) else 0
let $nooutputCount := if ($run/nooutput) then (if ($run/nooutput/@count) then
zero-or-one($run/nooutput/@count) else 0) + count($run/nooutput/task) else 0
let $totalTasks := $successCount + $failureCount + $nooutputCount
return <run>
{ $run/@tool }
{ for $v in $run/prepared/date/@unixtime[. > 0] return attribute
preparation {$v - exactly-one($run/started/date/@unixtime)} }
{ for $v in $run/executed/date/@unixtime[. > 0] return attribute
execution {$v - exactly-one($run/prepared/date/@unixtime)} }
{ for $v in $run/validated/date/@unixtime[. > 0] return attribute
validation {$v - exactly-one($run/executed/date/@unixtime)} }
{ for $v in $run/committed/date/@unixtime[. > 0] return attribute
commit {$v - exactly-one($run/validated/date/@unixtime)} }
{ for $v in $run/prepared/date/@unixtime[. > 0 and $totalTasks > 0]
return attribute preparationPerTask {($v -
exactly-one($run/started/date/@unixtime)) div $totalTasks} }
{ for $v in $run/executed/date/@unixtime[. > 0 and $totalTasks > 0]
return attribute executionPerTask {($v -
exactly-one($run/prepared/date/@unixtime)) div $totalTasks} }
{ for $v in $run/validated/date/@unixtime[. > 0 and $totalTasks > 0]
return attribute validationPerTask {($v -
exactly-one($run/executed/date/@unixtime)) div $totalTasks} }
{ for $v in $run/committed/date/@unixtime[. > 0 and $totalTasks > 0]
return attribute commitPerTask {($v -
exactly-one($run/validated/date/@unixtime)) div $totalTasks} }
{attribute allowedExecution {exactly-one($run/allowedExecutionTime)
* 1000}}
{attribute successCount {$successCount}}
{attribute failureCount {$failureCount}}
{attribute nooutputCount {$nooutputCount}}
{attribute totalTasks {$totalTasks}}
</run>
but, as you might have noticed the query is not written in a very nice way...
Therefore i tried to rewrite the query to the (hopefully) equivalent:
for $run in pf:collection("log.xml")//run
let $successCount := count($run/success/task)
let $failureCount := count($run/failure/task)
let $nooutputCount := if ($run/nooutput/@count) then
exactly-one($run/nooutput/@count) cast as xs:integer else 0 +
count($run/nooutput/task)
let $totalTasks := ($successCount + $failureCount + $nooutputCount) cast as
xs:integer
let $started := $run/started/date/@unixtime cast as xs:integer?
let $prepared := $run/prepared/date/@unixtime cast as xs:integer?
let $executed := $run/executed/date/@unixtime cast as xs:integer?
let $validated := $run/validated/date/@unixtime cast as xs:integer?
let $committed := $run/committed/date/@unixtime cast as xs:integer?
return element { "run" } {
$run/@tool,
if ($prepared > 0) then attribute preparation { $prepared - $started } else
(),
if ($executed > 0) then attribute execution { $executed - $prepared } else
(),
if ($validated > 0) then attribute validation { $validated - $executed }
else (),
if ($committed > 0) then attribute commit { $committed - $validated } else
(),
if ($totalTasks > 0)
then
(if ($prepared > 0) then attribute preparationPerTask { ($prepared -
$started) div $totalTasks } else (),
if ($executed > 0) then attribute executionPerTask { ($executed -
$prepared) div $totalTasks } else (),
if ($validated > 0) then attribute vaalidationPerTask { ($validated -
$executed) div $totalTasks } else (),
if ($committed > 0) then attribute commitPerTask { ($committed -
$validated) div $totalTasks } else ())
else
(),
attribute allowedExecution {exactly-one($run/allowedExecutionTime) * 1000},
attribute successCount {$successCount},
attribute failureCount {$failureCount},
attribute nooutputCount {$nooutputCount},
attribute totalTasks {$totalTasks}
}
This query compiles within seconds, but then the runtime environment complains:
MAPI = mone...@localhost
QUERY = for $run in pf:collection("log.xml")//run
ERROR = !ERROR: merged_union: tail of first BAT/column of table 2 must be
sorted.
!ERROR: CMDmerged_union: operation failed.
Unfortunately I cannot provide the log.xml-document, but when leaving out the
if-statement:
"if ($totalTasks > 0) then ... else ()"
the query executes fine.
p.s. This workaround works for us at the moment.
----------------------------------------------------------------------
>Comment By: Wouter Alink (vzzzbx)
Date: 2009-11-12 19:09
Message:
cant get access to the data, closing bug
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-02-15 22:09
Message:
ALG compilation (also) of the first query (now) seems to work fine.
Wouter,
could you please either provide the data to make a proper test (also) for
the reported (and fixed) merged_union problem, or close this bug report
with a comment that (and why) a test is not possible or required?
Thanks!
Stefan
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-12-12 00:54
Message:
I don't have the data to re-produce this bug;
if at all, the original submitter (Wouter) needs to either provide us
should with the respective data to add a test, or check (and accept) the
fix without test and close the bug report.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-06-20 09:04
Message:
Logged In: YES
user_id=572415
Originator: NO
Added tests (compilation, only) in
pathfinder/tests/BugTracker/Tests/compilation_or_runtime_choice.SF-1991738_[01]_{MPS,ALG}:
- First query compiles fine with MPS.
- First query takes >30 minutes & >6 GB to compile with ALG.
- Second query fails to compile with MPS:
type error: can't cast type 'untypedAtomic*' to type 'integer?'
- Second query compiles fine with ALG.
A test to trigger the (now fixed) merged_union problem with the second
query is still pending --- mainly, some proper data is missing ...
----------------------------------------------------------------------
Comment By: Jan Rittinger (tsheyar)
Date: 2008-06-16 16:47
Message:
Logged In: YES
user_id=993208
Originator: NO
The merged_union problem is fixed in CVS. A test triggering this problem
is still pending.
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2008-06-16 16:22
Message:
Logged In: YES
user_id=621590
Originator: YES
yep, and when doing the same for the merged_union at lines (+-) 2090,
2512, 2765, 3190, 3444, 3873
then the query returns an answer that looks like the expected result.
----------------------------------------------------------------------
Comment By: Jan Rittinger (tsheyar)
Date: 2008-06-16 13:48
Message:
Logged In: YES
user_id=993208
Originator: NO
Wouter, Am I right that adding the line
assert_order(a0090);
before the failing merged_union fixes the problem for this call to
merged_union?
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2008-06-16 12:54
Message:
Logged In: YES
user_id=621590
Originator: YES
The error:
ERROR = !ERROR: merged_union: tail of first BAT/column of table 2 must be
sorted.
!ERROR: CMDmerged_union: operation failed.
happens in the merged_union on line 1854:
a0051 := unused;
a0007 := unused;
a0007 := [lng](a0039);
if (a0007.reverse ().exist (lng(nil))) {
ERROR ("err:FORG0001.");
} else {
}
a0051 := unused;
a0051 := [+](a0107, a0108);
a0107 := unused;
a0108 := unused;
a0039 := unused;
a0039 := merged_union (a0103, a0090, a0097, a0075, a0098, a0076,
a0099, a0077, a0100, a0087, a0101, a0088, a0102, a0089, a0007, a0047,
a0051, a0050);
a0056 := a0039.fetch (0);
a0057 := a0039.fetch (1);
a0058 := a0039.fetch (2);
a0059 := a0039.fetch (3);
a0060 := a0039.fetch (4);
a0073 := a0039.fetch (6);
a0074 := a0039.fetch (7);
a0091 := a0039.fetch (8);
a0039 := unused;
a0103 := unused;
a0097 := unused;
a0098 := unused;
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2008-06-13 18:53
Message:
Logged In: YES
user_id=621590
Originator: YES
Hello Jan,
There is not much that I expect out of this bugreport. We found a viable
workaround for the given query, so there is certainly no hurry in fixing
this. I wanted to indicate that there were problems with running this
query. I will try to provide the failing line in the milscript on monday.
For the first query: i knew monetdb was memory-hungry, however it might be
nice to have some 'detection-mechanism' that stops optimizing when more
than 2?GB of memory is spent by the pathfinder compiler alone trying to
come up with a query plan... but i guess that is probably a feature request
for a next release :)
----------------------------------------------------------------------
Comment By: Jan Rittinger (tsheyar)
Date: 2008-06-13 09:42
Message:
Logged In: YES
user_id=993208
Originator: NO
Hi Wouter,
I'm not sure what you expect from this bug report.
The optimization of the first query screws up because the optimizer wants
to get rid of all ``for loops'' and has a harder time reaching this goal
than in the second case. If you however look at the query heavy
optimization is not needed to provide a good performance. Knowing this it
could be perhaps helpful to state that you only want to apply ``light''
optimization for this query (in comparison to others).
For the second query I cannot see anything wrong in the code generation.
To solve this problem I would need some help from you.
If you generate the query with pf -A, profile it, give the position in the
MIL code (the merged_union that is going to fail; probably one of the 3
calls with 10 arguments), and perhaps a characteristic snippet of the
second input BAT (it should be an iter/oid column), then it should be
possible to decide whether the underlying string_join() operator or an
incorrect property inference in MonetDB might be the reason for the failing
merged_union() call.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1991738&group_id=56967
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
