Danny, I suspect the server is being smart about not streaming out the entire result set if you are getting .029 seconds for 88,000 documents but not really accessing the document sub-elements, or they are all in memory - the expanded tree cache holds the fully retrieved documents until the cache fills up and they are ejected. When you start to access the document children or the database grows to a point that it needs to go back to disk because they are not cached you'll probably find worse performance.
If you are interested in fast analytics like sums and counts for a group you can use functions that only access lightweight data from a particular range index, like: cts:sum(), cts:element-values()+cts:frequency(), cts:average(); or you can get a random sample using cts:search(... "score-random") if exact numbers are not the point (which is almost always the case for human consumption of large data-sets, but won't be right for financial processing or other exact computations). If you describe your group-by requirement in more detail I can suggest a particular approach. Yours, Damon From: [email protected] [mailto:[email protected]] On Behalf Of Danny Sinang Sent: Wednesday, July 25, 2012 10:18 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] How to speed up getting child element Hi David, Thanks, but I need to fetch all the child elements of all the results to generate groupBy summaries. If i filter the results and then get the child elements, my groupBy summaries wont be accurate. Danny Sent from my iPhone On Jul 25, 2012, at 8:34 PM, David Lee <[email protected]<mailto:[email protected]>> wrote: First, it seems a bit odd you are using the search: api to create the search but not to execute it. You may find many of your problems solved by using search:search to execute the search. Not knowing what your search looks like I will guess you return documents based on a search then want the books out of them, where the search criteria may be outside the book element ? If so as others recommended use a subset of the results. But you need to do so in the same expression as the search or indexing is not as efficient. I.e instead of let $searchResults := cts:search(fn:collection("trustedPartnerLogs"), cts:query($cts-query)) return $searchResults/book Do let $searchResults := cts:search(fn:collection("trustedPartnerLogs"), cts:query($cts-query))[1 to 100]/books Or if need be let $searchResults := (cts:search(fn:collection("trustedPartnerLogs"), cts:query($cts-query))/books)[1 to 100] Generally the first rule is you dont want to load into memory thousands of documents to sub query with xpath. Rather refine the search to get closer to what you want or use a subset and then xpath those. Do so directly on the cts:search call not on a result variable. Better yet use the search:search API which can do much of this for you. ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected]<mailto:[email protected]> Phone: +1 650-287-2531 Cell: +1 812-630-7622 www.marklogic.com<http://www.marklogic.com> This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Danny Sinang Sent: Wednesday, July 25, 2012 4:22 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] How to speed up getting child element Hi David, Below is my cts:search() call. How do I do what you're suggesting ? let $cts-query := search:parse($searchQuery, $options) let $searchResults := cts:search(fn:collection("trustedPartnerLogs"), cts:query($cts-query)) Regards, Danny On Wed, Jul 25, 2012 at 3:01 PM, David Lee <[email protected]<mailto:[email protected]>> wrote: The other alternative (or in combination) ... if all you want is the book element , augment your cts:search to just return the book elements. ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected]<mailto:[email protected]> Phone: +1 650-287-2531<tel:%2B1%20650-287-2531> Cell: +1 812-630-7622<tel:%2B1%20812-630-7622> www.marklogic.com<http://www.marklogic.com> This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Geert Josten Sent: Wednesday, July 25, 2012 1:53 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] How to speed up getting child element Hi Danny, To put it simple: don't do that on all 88 thousand results, but only those you want to show on a search result page. That is why search:search applies pagination by default, quite convenient by the way, saves you from a lot of trouble doing it yourself.. Kind regards, Geert Van: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] Namens Danny Sinang Verzonden: woensdag 25 juli 2012 19:49 Aan: general Onderwerp: [MarkLogic Dev General] How to speed up getting child element Hi, I have a cts:search query that returns 88,542 results in juts 0.029 seconds. Problem is, when I do this : let $x := $searchResults/book the total elapsed time jumps to 2.87 seconds. Is there a better / faster way of doing this ? Regards, Danny _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
