Bugs item #2898944, was opened at 2009-11-17 09:27 Message generated for change (Comment added) made by stmane You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: PF/compiler Group: Pathfinder "candidate" Status: Open Resolution: Fixed Priority: 6 Private: No Submitted By: Axel Belinfante (axelbel) Assigned to: Jan Rittinger (tsheyar) Summary: pf takes 'forever' (> 1 hour) to compile particular query Initial Comment: I don't know whether the following really qualifies as bug, or just is a case of 'user error by giving stupid input. In any case, I would like to bring this to your attention. When I compile a query for attached module definition with pf (may or aug or pre-nov version), compilation takes 'forever'. When I replace the clause ecv:to-human($g0, $human_type) by simple $g0 and remove definitions of $human_type and to-human it compiles almost instantly. It is the huge difference in compilation time that surprises me. The query I compile is: import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv"; at "/tmp/long-compilation.xq"; ecv:view-from-save("eprints-export-conv-fmt.xml") where tmp/long-compilation.xq is the attached module. What the query attempts to do: obtain results from the database, order these on two criteria, and when listing the results, insert human-readable headers. The human-readable headers are obtained by the to-human mapping from field value (used as ordering criterium) to human-readable string. At some point I just used if-then-else-if-then-else etc. to do the mapping, but that seemed slower to compile than the approach I took here. Suggestions for rewriting the query to a fast-compiling one would be considered as a fix too, I guess. ---------------------------------------------------------------------- >Comment By: Stefan Manegold (stmane) Date: 2009-11-19 10:00 Message: Test works fine: http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.out.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.64.64.d.1-Fedora10/tests_BugTracker/slow_compilation.SF-2898944.err.00.html Axel, could you please check and report whether Jan's fix also works for you, and if so, close this bug report? Thanks! ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2009-11-18 19:35 Message: If this fix works out, it will be in the upcoming release. (We had some show stoppers with yesterday's release candidate.) ---------------------------------------------------------------------- Comment By: Stefan Manegold (stmane) Date: 2009-11-18 18:34 Message: =================================================================== 2009/11/18 - stmane: pathfinder/tests/BugTracker/Tests/All,1.146.2.11 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.bat,1.1.2.1 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod1.xq,1.1.2.1 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.mod2.xq,1.1.2.1 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q1.xq.in,1.1.2.1 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.q2.xq.in,1.1.2.1 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.sh,1.1.2.1 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.err,1.1.2.1 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.stable.out,1.1.2.1 pathfinder/tests/BugTracker/Tests/slow_compilation.SF-2898944.timeout,1.1.2.1 added test (compilation via pf, only) for ID: 2898944 "pf takes 'forever' (> 1 hour) to compile particular query" https://sourceforge.net/tracker/?func=detail&aid=2898944&group_id=56967&atid=482468 Seems to work fine, now; compiling both queries with both `pf -A` & `pf -M` takes less than 2 secs on my machine with a debug build. Set timeout to 6 secs. =================================================================== report can be closed, once nightly testing and Axel confirm the fix. ---------------------------------------------------------------------- Comment By: Jan Rittinger (tsheyar) Date: 2009-11-18 16:51 Message: Plan generation with permutations gets you into trouble. Planning the distinct operator requires a full ordering of the table as we use the order extend in MIL to remove duplicates. To calculate the plans for all possible orders we generate all permutations. (In the first long-compilation example distinct was applied to a table with 7 columns and for the second example distinct operated on 8 columns.) Now we first eliminate all functionally dependent and constant columns as they do not add any order information. If we still have too many permutations we ignore all orderings (plans) after a magic boundary. With this changes in place the compile times are (almost) acceptable: pf -T query-long-compilation.xq ... overall compilation time: 974ms pf -T query-long-compilation2.xq ... overall compilation time: 02s 781ms ---------------------------------------------------------------------- Comment By: Stefan Manegold (stmane) Date: 2009-11-18 12:53 Message: Just a few general notes: 1) A user program can at most trigger a machine crash, but never cause it. If a user program triggers a machine crash (e.g., by using resources (too?) many resources, it's in the end a problem (bug) in the OS (or the hardware) that cannot properly handle (or prevent) such (over-)load properly. 2) If you kill (via keyboard interrupt ^C or by sending, say, SIGTERM or SIGKILL) a process that uses lots of memory (and possibly virtual memory, i.e., swap), it might indeed take some time until that process is gone and your prompt is back, since the systems needs to clean up that memory (and swap) and possibly swap your shell back into main memory ... 3) Yes, a process that is using much memory (i.e., close to or more than the machine has physical memory) might (usually will) affect the interactive responsiveness of the machine. ---------------------------------------------------------------------- Comment By: Axel Belinfante (axelbel) Date: 2009-11-18 12:36 Message: the original long-compilation.xq now finishes in about 10 secs, which is still longer than I would like, but much shorter than it used to. regarding time measurements, I may have caused some confusion, in the sense that: - the original long-compilation.xq definitely took 'too long', but likely I always interruped it long before 1 hour passed - the query that prompted me change the bug summmary line to include '> 1 hour' was a different one, I'm sure that was long-compilation2.xq long-compilation2.xq might have taken down a machine before the fix (machine did crash while that pf was running, but we do not know whether that was the cause) after applying the fix and rerunning the pf on long-compilation2.xq I do notice that at some point the machine stops being responsive, only to become responsive again after the segfault - maybe the segfault saved us from a second machine crash :-) I wonder whether the memory consumption is what so badly affects responsivity of the machine. ---------------------------------------------------------------------- Comment By: Stefan Manegold (stmane) Date: 2009-11-18 12:03 Message: Axel, what about your original long-compilation.xq? Does that now work fine for you (i.e., with Jan's fix)? ---------------------------------------------------------------------- Comment By: Axel Belinfante (axelbel) Date: 2009-11-18 11:30 Message: I don't want to spoil the fun, but after the fix pf still runs pretty long on the file I just attached (long-compilation2.xq) top gives me: PID USER NI VIRT RES SHR P S %CPU %MEM TIME COMMAND 26940 dbguest 0 7164m 3.7g 172 4 R 27 96.5 14:07 /local/dbguest/Install/MonetDB-XQuery_Nov2009/bin/pf -T /tmp/talt11 I tried to kill it with ^C, without too much success (and I already started to fear that the machine was hung up on it), until a little later I saw (I started it as: time /path/to/pf xqueryfile) Killed 828.295u 39.446s 16:38.94 86.8% 0+0k 0+0io 5pf+0w Segmentation fault ---------------------------------------------------------------------- Comment By: Stefan Manegold (stmane) Date: 2009-11-18 11:03 Message: I think, adding a test (also) for this one should be done; it should be not only possible, but also easy: the query that (used to) compile for for than one hour is given, just make a test that calls pf to compile the query, sending the output to, say, /dev/null/ . If that finishes without error and within the default 1 minute timeout (or less, if desired/required; add a TSTNME.timeout file with a <1 scale factor), the test succeeded and everything is fine; otherwise, the test fails ... ... just an idea to help assessing and monitoring this issue and hence the stability and usability of MonetDB as such ... ... does not apply to this bug report, only; see also my email of last weekend ... ... if there is no time right no to add the test, keep the bug report open to remind us ... ... in fact, I do exactly that ... ---------------------------------------------------------------------- Comment By: Jan Rittinger (tsheyar) Date: 2009-11-18 10:20 Message: The reason for the 'forever' compile time is that the physical plan space blows up in a heuristic that tries to find good plan orders. I now added a check that avoids generating to many plans. I checked in the fix in the Nov2009 stable branch. I'm however not sure whether it makes it into the release -- probably not. So please get it from either CVS or nightly-builds. ---------------------------------------------------------------------- Comment By: Axel Belinfante (axelbel) Date: 2009-11-17 09:32 Message: not sure about category or group - chose seemingly most appropriate. even though I did not change default priority , having some answer for this (whether fix in code, or guideline for fast compiling query) would be very helpful. In my application the pf compilation run is triggered from a web-gui - fast compilation is very important there, and predicatable compilation times maybe even more. (i.e. it would be nice when all compilations would take the same order of time). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2898944&group_id=56967 ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Monetdb-bugs mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
