Bugs item #1991738, was opened at 2008-06-12 11:30
Message generated for change (Settings changed) made by boncz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1991738&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/compiler
Group: Pathfinder "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
>Assigned to: Stefan Manegold (stmane)
Summary: XQ: compilation or runtime choice
Initial Comment:
(stable nightly build, compiled from supersourcetarball, June 11, 2008)
The following query takes about 35 minutes to compile (and I have reason to
believe it was also the reason why the machine crashed after it was executed a
few times):
for $run in pf:collection("log.xml")//run
let $successCount := if ($run/success) then count($run/success/task) else 0
let $failureCount := if ($run/failure) then count($run/failure/task) else 0
let $nooutputCount := if ($run/nooutput) then (if ($run/nooutput/@count) then
zero-or-one($run/nooutput/@count) else 0) + count($run/nooutput/task) else 0
let $totalTasks := $successCount + $failureCount + $nooutputCount
return <run>
{ $run/@tool }
{ for $v in $run/prepared/date/@unixtime[. > 0] return attribute
preparation {$v - exactly-one($run/started/date/@unixtime)} }
{ for $v in $run/executed/date/@unixtime[. > 0] return attribute
execution {$v - exactly-one($run/prepared/date/@unixtime)} }
{ for $v in $run/validated/date/@unixtime[. > 0] return attribute
validation {$v - exactly-one($run/executed/date/@unixtime)} }
{ for $v in $run/committed/date/@unixtime[. > 0] return attribute
commit {$v - exactly-one($run/validated/date/@unixtime)} }
{ for $v in $run/prepared/date/@unixtime[. > 0 and $totalTasks > 0]
return attribute preparationPerTask {($v -
exactly-one($run/started/date/@unixtime)) div $totalTasks} }
{ for $v in $run/executed/date/@unixtime[. > 0 and $totalTasks > 0]
return attribute executionPerTask {($v -
exactly-one($run/prepared/date/@unixtime)) div $totalTasks} }
{ for $v in $run/validated/date/@unixtime[. > 0 and $totalTasks > 0]
return attribute validationPerTask {($v -
exactly-one($run/executed/date/@unixtime)) div $totalTasks} }
{ for $v in $run/committed/date/@unixtime[. > 0 and $totalTasks > 0]
return attribute commitPerTask {($v -
exactly-one($run/validated/date/@unixtime)) div $totalTasks} }
{attribute allowedExecution {exactly-one($run/allowedExecutionTime)
* 1000}}
{attribute successCount {$successCount}}
{attribute failureCount {$failureCount}}
{attribute nooutputCount {$nooutputCount}}
{attribute totalTasks {$totalTasks}}
</run>
but, as you might have noticed the query is not written in a very nice way...
Therefore i tried to rewrite the query to the (hopefully) equivalent:
for $run in pf:collection("log.xml")//run
let $successCount := count($run/success/task)
let $failureCount := count($run/failure/task)
let $nooutputCount := if ($run/nooutput/@count) then
exactly-one($run/nooutput/@count) cast as xs:integer else 0 +
count($run/nooutput/task)
let $totalTasks := ($successCount + $failureCount + $nooutputCount) cast as
xs:integer
let $started := $run/started/date/@unixtime cast as xs:integer?
let $prepared := $run/prepared/date/@unixtime cast as xs:integer?
let $executed := $run/executed/date/@unixtime cast as xs:integer?
let $validated := $run/validated/date/@unixtime cast as xs:integer?
let $committed := $run/committed/date/@unixtime cast as xs:integer?
return element { "run" } {
$run/@tool,
if ($prepared > 0) then attribute preparation { $prepared - $started } else
(),
if ($executed > 0) then attribute execution { $executed - $prepared } else
(),
if ($validated > 0) then attribute validation { $validated - $executed }
else (),
if ($committed > 0) then attribute commit { $committed - $validated } else
(),
if ($totalTasks > 0)
then
(if ($prepared > 0) then attribute preparationPerTask { ($prepared -
$started) div $totalTasks } else (),
if ($executed > 0) then attribute executionPerTask { ($executed -
$prepared) div $totalTasks } else (),
if ($validated > 0) then attribute vaalidationPerTask { ($validated -
$executed) div $totalTasks } else (),
if ($committed > 0) then attribute commitPerTask { ($committed -
$validated) div $totalTasks } else ())
else
(),
attribute allowedExecution {exactly-one($run/allowedExecutionTime) * 1000},
attribute successCount {$successCount},
attribute failureCount {$failureCount},
attribute nooutputCount {$nooutputCount},
attribute totalTasks {$totalTasks}
}
This query compiles within seconds, but then the runtime environment complains:
MAPI = mone...@localhost
QUERY = for $run in pf:collection("log.xml")//run
ERROR = !ERROR: merged_union: tail of first BAT/column of table 2 must be
sorted.
!ERROR: CMDmerged_union: operation failed.
Unfortunately I cannot provide the log.xml-document, but when leaving out the
if-statement:
"if ($totalTasks > 0) then ... else ()"
the query executes fine.
p.s. This workaround works for us at the moment.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-06-20 09:04
Message:
Logged In: YES
user_id=572415
Originator: NO
Added tests (compilation, only) in
pathfinder/tests/BugTracker/Tests/compilation_or_runtime_choice.SF-1991738_[01]_{MPS,ALG}:
- First query compiles fine with MPS.
- First query takes >30 minutes & >6 GB to compile with ALG.
- Second query fails to compile with MPS:
type error: can't cast type 'untypedAtomic*' to type 'integer?'
- Second query compiles fine with ALG.
A test to trigger the (now fixed) merged_union problem with the second
query is still pending --- mainly, some proper data is missing ...
----------------------------------------------------------------------
Comment By: Jan Rittinger (tsheyar)
Date: 2008-06-16 16:47
Message:
Logged In: YES
user_id=993208
Originator: NO
The merged_union problem is fixed in CVS. A test triggering this problem
is still pending.
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2008-06-16 16:22
Message:
Logged In: YES
user_id=621590
Originator: YES
yep, and when doing the same for the merged_union at lines (+-) 2090,
2512, 2765, 3190, 3444, 3873
then the query returns an answer that looks like the expected result.
----------------------------------------------------------------------
Comment By: Jan Rittinger (tsheyar)
Date: 2008-06-16 13:48
Message:
Logged In: YES
user_id=993208
Originator: NO
Wouter, Am I right that adding the line
assert_order(a0090);
before the failing merged_union fixes the problem for this call to
merged_union?
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2008-06-16 12:54
Message:
Logged In: YES
user_id=621590
Originator: YES
The error:
ERROR = !ERROR: merged_union: tail of first BAT/column of table 2 must be
sorted.
!ERROR: CMDmerged_union: operation failed.
happens in the merged_union on line 1854:
a0051 := unused;
a0007 := unused;
a0007 := [lng](a0039);
if (a0007.reverse ().exist (lng(nil))) {
ERROR ("err:FORG0001.");
} else {
}
a0051 := unused;
a0051 := [+](a0107, a0108);
a0107 := unused;
a0108 := unused;
a0039 := unused;
a0039 := merged_union (a0103, a0090, a0097, a0075, a0098, a0076,
a0099, a0077, a0100, a0087, a0101, a0088, a0102, a0089, a0007, a0047,
a0051, a0050);
a0056 := a0039.fetch (0);
a0057 := a0039.fetch (1);
a0058 := a0039.fetch (2);
a0059 := a0039.fetch (3);
a0060 := a0039.fetch (4);
a0073 := a0039.fetch (6);
a0074 := a0039.fetch (7);
a0091 := a0039.fetch (8);
a0039 := unused;
a0103 := unused;
a0097 := unused;
a0098 := unused;
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2008-06-13 18:53
Message:
Logged In: YES
user_id=621590
Originator: YES
Hello Jan,
There is not much that I expect out of this bugreport. We found a viable
workaround for the given query, so there is certainly no hurry in fixing
this. I wanted to indicate that there were problems with running this
query. I will try to provide the failing line in the milscript on monday.
For the first query: i knew monetdb was memory-hungry, however it might be
nice to have some 'detection-mechanism' that stops optimizing when more
than 2?GB of memory is spent by the pathfinder compiler alone trying to
come up with a query plan... but i guess that is probably a feature request
for a next release :)
----------------------------------------------------------------------
Comment By: Jan Rittinger (tsheyar)
Date: 2008-06-13 09:42
Message:
Logged In: YES
user_id=993208
Originator: NO
Hi Wouter,
I'm not sure what you expect from this bug report.
The optimization of the first query screws up because the optimizer wants
to get rid of all ``for loops'' and has a harder time reaching this goal
than in the second case. If you however look at the query heavy
optimization is not needed to provide a good performance. Knowing this it
could be perhaps helpful to state that you only want to apply ``light''
optimization for this query (in comparison to others).
For the second query I cannot see anything wrong in the code generation.
To solve this problem I would need some help from you.
If you generate the query with pf -A, profile it, give the position in the
MIL code (the merged_union that is going to fail; probably one of the 3
calls with 10 arguments), and perhaps a characteristic snippet of the
second input BAT (it should be an iter/oid column), then it should be
possible to decide whether the underlying string_join() operator or an
incorrect property inference in MonetDB might be the reason for the failing
merged_union() call.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1991738&group_id=56967
------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
