The size and structure of the query can matter a lot. Michael's example shows 1000 query parts, but you'll see a linear growth if you let it iterate up to 20000. On the other hand, if you pass in those 20k random id's as one large sequence of allowed values into one range query, the profile time drops to about 50msec again.. :)
@Ron, can you make the query smarter? Doing the same with less parts? Kind regards, Geert > -----Oorspronkelijk bericht----- > Van: [email protected] [mailto:general- > [email protected]] Namens Michael Blakeley > Verzonden: woensdag 31 juli 2013 20:20 > Aan: MarkLogic Developer Discussion > Onderwerp: Re: [MarkLogic Dev General] Registered Query Best Practices > > If that profile is correct, I'd be much more worried about the cts:query > constructor. I have a hard time getting that much elapsed time out of a > cts:query constructor. > > declare variable $q := cts:and-query( > (1 to 1000) ! cts:element-range-query( > xs:QName('does-not-exist'), '=', > xdmp:integer-to-hex(xdmp:random()))) ; > > prof:eval(' > declare variable $qnode as element() external ; > cts:query($qnode)', > (xs:QName('qnode'), > document { $q }/*)) > > On my laptop, the profile shows cts:query at 99% of shallow, with > anywhere from 5-15 ms total. And that's with 1000 terms, which seems like > a lot. But I'm testing against a nearly empty database, which might make a > difference. > > Is cts:query still a hotspot if you drop the registered-query code entirely? > > Is there a particular cts:query term type that triggers this? > > Does xdmp:query-meters() show anything indicating database lookups? > > -- Mike > > On 31 Jul 2013, at 09:37 , Ron Hitchens <[email protected]> wrote: > > > > > So here's a little more color on this, if anyone is still > > interested. When I profile this code, where $query is a fairly > > complex serialized query that was previously computed and stored > > in a database: > > > > declare variable $q1 := cts:registered-query (cts:register (cts:query > ($query)), "unfiltered"); > > > > cts:search (fn:doc(), $q1)[1 to 5] > > > > The top two items on the profile output are: > > > > Shallow% Shallow usecs Deep% Deep usecs Expression > > 80 125000 90 140000 cts:query($query) > > 10 16000 100 156000 cts:registered-query (cts:register > (cts:query ($query)), "unfiltered") > > > > Time spent on the actual search is so small it rounded to zero. > > > > Doing this repeatedly yields similar timing, so it's not a cold > > cache situation or anything like that. > > > > Profiling this: > > > > declare variable $q2 := cts:registered-query (9156609332438599120, > "unfiltered"); > > > > cts:search (fn:doc(), $q2)[1 to 5] > > > > Yields times too fast to measure (all rounded to zero) > > > > So, the potentially expensive to create query is being > > built every time and possibly being re-registered as well, > > given that cts:registered-query is taking a non-trivial amount > > of time. > > > > On Jul 31, 2013, at 8:38 AM, Ron Hitchens <[email protected]> wrote: > > > >> > >> The overall entitlement query on each request is composed > >> of many sub-queries, some of which are static and registered, > >> some of which are dependent on the current time. But even the > >> static ones are not finite, new ones can be created at any time > >> as part of a new entitlement definition. > >> > >> I'm working on a scheme to catch and re-register all the > >> static queries in a given query tree when a search fails due > >> to a missing registration. That should lazily re-register > >> on first use after a server restart as well. > >> > >> --- > >> Ron Hitchens {mailto:[email protected]} Ronsoft Technologies > >> +44 7879 358 212 (voice) http://www.ronsoft.com > >> +1 707 924 3878 (fax) Bit Twiddling At Its Finest > >> "No amount of belief establishes any fact." -Unknown > >> > >> > >> On Jul 30, 2013, at 8:30 PM, Geert Josten <[email protected]> > wrote: > >> > >>> Hi Ron, > >>> > >>> Are your queries such that you would have a finite number of sub- > queries, > >>> if you would break them into smaller subparts? Perhaps you can > combine > >>> multiple registered queries.. > >>> > >>> Cheers, > >>> Geert > >>> > >>>> -----Oorspronkelijk bericht----- > >>>> Van: [email protected] [mailto:general- > >>>> [email protected]] Namens Ron Hitchens > >>>> Verzonden: dinsdag 30 juli 2013 2:29 > >>>> Aan: MarkLogic Developer Discussion > >>>> Onderwerp: Re: [MarkLogic Dev General] Registered Query Best > Practices > >>>> > >>>> > >>>> Hi Geert, > >>>> > >>>> I've done something before where we stored reg ids in a map for > >>>> easy re-use. In that case, there was a 1:1 correspondence between > >>>> the reg id and a meaningful business domain number. On this project > >>>> that's not the case. > >>>> > >>>> Also, there is not a finite set of queries that need to be registered > >>>> so it's not feasible to pre-register everything once. New ones can be > >>>> created > >>>> dynamically. And the complicated queries are persisted in another > >>>> database > >>>> and can be referenced later. This means the queries which should be > >>>> registered > >>>> will persist across server restarts. Which means there must be a way > to > >>>> register the queries on first use, then make use of those registered > >>> queries > >>>> on subsequent requests. > >>>> > >>>> The re-register-before-each-use pattern solves that nicely, but not > >>> if > >>>> the query construction cost must be re-paid each time. It looks like > >>> the > >>>> robust solution is going to have to be catching exceptions for > >>> unregistered > >>>> queries and reconstructing the registrations. It's a shame because > that > >>> is > >>>> going to add unnecessary complexity to the code. > >>>> > >>>> --- > >>>> Ron Hitchens {mailto:[email protected]} Ronsoft Technologies > >>>> +44 7879 358 212 (voice) http://www.ronsoft.com > >>>> +1 707 924 3878 (fax) Bit Twiddling At Its Finest > >>>> "No amount of belief establishes any fact." -Unknown > >>>> > >>>> > >>>> On Jul 29, 2013, at 8:15 PM, Geert Josten <[email protected]> > wrote: > >>>> > >>>>> Hi Ron, > >>>>> > >>>>> I recently saw a strategy where they deliberately took a different > >>>>> approach. In their case the calculation of the queries was not > >>>>> straight-forward and could run into 30k search terms. Additionally, > >>>>> registering the query, and warming up cache by doing one initial > >>> search > >>>>> after registering each query took most time. They were searching > >>> roughly > >>>>> 40mln docs. The searches themselves were subsec.. > >>>>> > >>>>> Their approach was to store all registered query id's somewhere, and > >>> have > >>>>> them readily available at actual search time. They also used a try > >>> catch > >>>>> to catch unregistered queries, though in their case they shouldn't > >>>>> actually occur, and these dramatically pulled down the average on > >>>>> performance tests. > >>>>> > >>>>> How much chance is there that a query is unregistered, if you would > >>>>> prepare all queries beforehand? > >>>>> > >>>>> Cheers, > >>>>> Geert > >>>>> > >>>>>> -----Oorspronkelijk bericht----- > >>>>>> Van: [email protected] [mailto:general- > >>>>>> [email protected]] Namens Michael Blakeley > >>>>>> Verzonden: maandag 29 juli 2013 21:08 > >>>>>> Aan: MarkLogic Developer Discussion > >>>>>> Onderwerp: Re: [MarkLogic Dev General] Registered Query Best > >>> Practices > >>>>>> > >>>>>> I think you're using registered query as intended. That behavior > >>> sounds > >>>>> odd > >>>>>> to me. I would expect (2) to be cheap, just a hash operation on the > >>>>> query > >>>>>> terms, and I would (3) to be the expensive step. > >>>>>> > >>>>>> So I would contact support and see what they think. > >>>>>> > >>>>>> -- Mike > >>>>>> > >>>>>> On 29 Jul 2013, at 11:03 , Ron Hitchens <[email protected]> wrote: > >>>>>> > >>>>>>> > >>>>>>> What is the best practice these days for using registered > >>>>>>> queries? I was under the impression that the pattern should be: > >>>>>>> > >>>>>>> 1) Create your query: > >>>>>>> $query := cts:and-query ((blah blah blah)) > >>>>>>> 2) Register it and make a registered query from it in one step: > >>>>>>> $reg-query := cts:resistered-query (cts:register ($query), > >>>>> "unfiltered") > >>>>>>> 3) Use it in a search: > >>>>>>> cts:search (fn:doc(), $reg-query) > >>>>>>> > >>>>>>> The theory being that if the cts:query described by $query is > >>>>>>> already registered, then the registration is essentially a no-op > >>>>>>> and you'll get back the same ID. And doing this every time insures > >>>>>>> that if the registered query has been evicted for some reason then > >>>>>>> it's re-registered and all is well. > >>>>>>> > >>>>>>> It's a nice theory but seems to be based on the assumption that > >>>>>>> creating a cts:query object is very cheap. Unfortunately, I'm > >>> finding > >>>>>>> that this is often not the case, especially when there are lots of > >>>>>>> documents in the database. I have a test case where performing > Step > >>> 2 > >>>>>>> above on a moderately complicated query takes roughly 200ms > every > >>>>>> time. > >>>>>>> Others take even longer and all seem to be proportional to > database > >>>>> size. > >>>>>>> But running Step 3 with cts:registered-query(<regid>) is very, very > >>>>>>> fast (~0ms). Re-creating the query for re-registering every time is > >>>>>>> destroying the benefit of using a registered query. > >>>>>>> > >>>>>>> I can obviously save the registration ID obtained from calling > >>>>>>> cts:register and then make a cts:registered-query each time, but > >>> then > >>>>>>> I'm not protected from the query becoming unregistered. And > there > >>> is > >>>>>>> no lightweight way to test if an ID is still registered. The only > >>> way > >>>>>>> I know to make this robust is to put a loop and try/catch around > the > >>>>>>> code that does the search. But that requires passing along > enough > >>>>>>> context to re-construct and re-register the queries (there can be > >>>>>>> dozens of them in this case). This is obviously a lot harder than > >>>>>>> building the complex query in one module and then passing it > along > >>>>>>> to the search code somewhere else. > >>>>>>> > >>>>>>> What's the generally accepted best usage pattern for registered > >>>>>>> queries? And is it my imagination or has the cost of running > >>> queries > >>>>>>> been moving from query evaluation into query construction? > >>>>>>> > >>>>>>> Thanks. > >>>>>>> > >>>>>>> --- > >>>>>>> Ron Hitchens {mailto:[email protected]} Ronsoft Technologies > >>>>>>> +44 7879 358 212 (voice) http://www.ronsoft.com > >>>>>>> +1 707 924 3878 (fax) Bit Twiddling At Its Finest > >>>>>>> "No amount of belief establishes any fact." -Unknown > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> General mailing list > >>>>>>> [email protected] > >>>>>>> http://developer.marklogic.com/mailman/listinfo/general > >>>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> General mailing list > >>>>>> [email protected] > >>>>>> http://developer.marklogic.com/mailman/listinfo/general > >>>>> _______________________________________________ > >>>>> General mailing list > >>>>> [email protected] > >>>>> http://developer.marklogic.com/mailman/listinfo/general > >>>> > >>>> _______________________________________________ > >>>> General mailing list > >>>> [email protected] > >>>> http://developer.marklogic.com/mailman/listinfo/general > >>> _______________________________________________ > >>> General mailing list > >>> [email protected] > >>> http://developer.marklogic.com/mailman/listinfo/general > >> > > > > _______________________________________________ > > General mailing list > > [email protected] > > http://developer.marklogic.com/mailman/listinfo/general > > > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
