Bugs item #2859986, was opened at 2009-09-16 15:24 Message generated for change (Comment added) made by axelbel You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2859986&group_id=56967
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: PF/runtime Group: Pathfinder "stable" Status: Open Resolution: None Priority: 5 Private: No Submitted By: Axel Belinfante (axelbel) Assigned to: Ying Zhang (yingying) Summary: xrpc use messes up mclient use Initial Comment: doing an xquery via mclient works immediately after starting Monetdb. doing then a couple of xrpc requests somehow messes up some state in the server, such that then doing the same xquery request via mclient once more yields an ERROR result, or an empty result. I have seen cases where at some point mclient continuously returns ERROR for all xquery queries, even for simple ones like <p>aap</p> . in other cases I just get an empty result. it is easy to reproduce on our system. but turned out to be harder to isolate such that it still is reproducable. I hope I succeeded in that. server: Aug2009 Monetdb release, running on (uname -a output): Linux ewi865 2.6.16.21-0.8-smp #1 SMP Mon Jul 3 18:25:39 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux ---------------------------------------------------------------------- >Comment By: Axel Belinfante (axelbel) Date: 2009-09-27 23:07 Message: Thanks for your efforts to reproduce and explain the problem. Having read it, I wonder whether this is related to what I reported as issue 2864991 ( https://sourceforge.net/tracker/?func=detail&aid=2864991&group_id=56967&atid=482468 ) I'll have to think on how to proceed next. I had already tried to work around the problem by using mapin instead of xrpc, (which I where I encountered that problem 2864991, which I then reproduced using only mclient). However, from your explanation I gather that, as long as I continue to use queries that call a module function with only (a single) string parameter(s), I will continue to encounter the problem, because pathfinder will try to use the caching. to give some background: ", building a web application (via ruby-on-rails) that allows a user to construct queries. these queries are saved in abstract form of xml, as well as xquery module. subsequently, these saved xqueries are used via a cgi script. performance (speed) is important to us, especially in the last part: the use of queries via cgi. ideally then they should appear to be as fast as an ordinary html page request. however, also when the user is constructing queries, and trying these out, performance matters, to give the user a responsive application. we (designers of the application) have been discussing the idea of using pf to compile a query to when we save it in the web-application, such that execution of the pre-compiled query can be very fast (both when it is run from the web-app, as well as via the cgi script) after this long intro, the question: would such compilation help to avoid the issue? (sidenote: when I tried to compile queries with the May 2009 release, I encountered cases where the compiler would appear to run forever; I did not yet retry with the latest release, but from release notes got the impression that in this respect this should have improved?) ---------------------------------------------------------------------- Comment By: Ying Zhang (yingying) Date: 2009-09-25 14:20 Message: Hello Axel, I have run your queries, and I can reproduce it (the xrpc request messages were first translated into xquery queries with xrpc calls). I think it is a bug in the function cache in MPS. Function cache in pathfinder: in the (old) MPS front-end, the whole module file is compiled, when it is 'import'-ed by a query. The compiler output of each function (i.e., the MIL-tree) in the module file is cached, so that a subsequent call to a function in the same module could be compiled (much) faster. So, in an Mserver session, each module file is only compiled once (inline, user-defined-functions are not cached). Function cache only works with MPS front-end, not with the (new) algebra front-end. For all functions called over XRPC, function cache is used. For normal queries (i.e., without any xrpc statement), function cache is used, if we can detect that the query contains a *single* call to a predefined function, with *only* simple literal paramters (i.e., each parameter is a single atomic-value). In Pathfinder runtime, we always check if function cache could be used, even if mclient has been started to use the algebra front-end. Once we found out that a query could be handeled with cached function, we automatically switch to the MPS front-end. And this is what has happend to your mclient-query06.xq, since it calls one module function with a single string parameter. Hence, both your normal query and your xrpc queries use the same cached MIL-trees, and most probably, the problem is caused by a not completely/correctly cleaned-up environment. I don't know how to fix this problem yet. I only have some suggestions as workaround of this problem. I hope they could be helpful for you, so that you could continue your work. Basically, what we want to do here, is to prevent pathfinder from using function cache for your normal query. This could be done in different ways (I have tried both and it seems working): 1. do not use module function in your normal query, but directly inline the function bodies in the query. For instance, I have replaced function calls in mclient-query06.xq as the following: $ cat mclient-query06.xq (: import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv"; at "/ufs/zhang/tmp/axelbel/ECVtest/xrpc/export/dyn/repo/dynxquery/axel/zambon/0"; ecv:view-from-save("eprints-export-conv-fmt.xml") :) let $eps := for $ep in doc("eprints-export-conv-fmt.xml")//tuple where ( ( ( ( some $c in $ep//creators_family satisfies contains(upper-case($c), "ZAMBON")) ) ) ) return $ep return <div>{ <div class="items">{$eps//fmt/div}</div> }</div> advantage: query compilation time is short (~30ms with MPS, ~20ms with algebra), as no unused functions are compiled. disadvantage: query maintainance, as you must update the normal query, if the corresponding functions in the module file have been changed. 2. this is very hacky: pass more complex parameter to the function call, so that it cannot be detected by the pathfinder runtime, as for which function cache could be used. This could be done very easily, for example: import module namespace ecv = "http://www.cs.utwente.nl/~belinfan/ecv"; at "/ufs/zhang/tmp/axelbel/ECVtest/xrpc/export/dyn/repo/dynxquery/axel/zambon/0"; ecv:view-from-save( ("eprints-export-conv-fmt.xml") ) ^ ^ Note the extra parentheses, now the parameter has become a sequence containing one string value. It is still valid, but will prevent the pathfinder runtime from using function cache :P advantage: ease of maintainance, minimal change of code. disadvantage: long compile time! For each normal query, the complete module must be load and compiled. With algebra, this costs ~300ms; with MPS, ~700ms. Would you please let us know if you can live with any of my suggestions, so that we can increase the priority of this bug accordingly? Regards, Jennie ---------------------------------------------------------------------- Comment By: Ying Zhang (yingying) Date: 2009-09-17 18:10 Message: Hello Axel, Thank you very much for reporting this problem. In the very beginning when xrpc was implemented, we have notices that there are situations, in which xrpc conflicts with mclient. We have tried to solve the cases we have noticed, but we could have missed some cases. I will look into your problem soon and post more information here, as soon as I know more. Regards, Jennie ---------------------------------------------------------------------- Comment By: Axel Belinfante (axelbel) Date: 2009-09-17 10:11 Message: I wrote: "however, if I then do the same request via xrpc, by repeating the third xrpc request, I do get a correct result..." this is not always the case. sometimes then such xrpc request returns an empty result ( <div><div class="items"/></div> ). Sometimes when the mclient request fails, it returns 'ERROR', together with almost the correct result: the result is correct, but the first couple of characters are missing (overwritten by the ERROR?). Finally, I don't know whether it is related to this issue, it happens that the server gets into a state where a pf:add-doc that is done via mclient just hangs, forever, until I kill the server to restart it. This issue was already present in the May-2009 release, and is still present in the Aug-2009 release. ---------------------------------------------------------------------- Comment By: Axel Belinfante (axelbel) Date: 2009-09-16 15:42 Message: I atdded the files that I used to set up the database at http://library.cs.utwente.nl/ecv/files-for-bugreport-2859986.tgz (size: 22861872 bytes) the file start-mxq-test.mil in the .tgz shows how I initialized the database. I invoked the xrpc commands by using telnet from within an xterm, and pasting the contents of the attached files. I invoked mclient as: mclient -lx -p myportnumber mclient-query06 so, when I cleanup the database 'var' directory, and restart it the mclient query works fine after I have done the first xrpc query, the mclient query still works fine after I have done the second xrpc query, the mclient query still works fine after I have done the third xrpc query, the mclient query returns an 'empty' result ( <div><div class="items"/></div> ) however, if I then do the same request via xrpc, by repeating the third xrpc request, I do get a correct result... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2859986&group_id=56967 ------------------------------------------------------------------------------ Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf _______________________________________________ Monetdb-bugs mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
