An example of what Michael said would be use element values with option frequency order and cts:frequency. You might be assuming we can't do something that we are perfectly optimized to do.
If that's not the case the recommendations on spawn and corb make sense. As for spawn with an update: just remember spawn won't be rolled back and that might be why the documentation says that. Ideally you should have a query statement that spawns update statements. Makes sense? Nuno On Jun 2, 2011 3:06 PM, "Michael Blakeley" <[email protected]> wrote: > Steve, the suggestions related to spawn and limits are good, but you might want to back up and reconsider the problem. Naturally you know the details of your problem best, but it might be possible to do some or all of the work more efficiently using existing product features. > > -- Mike > > On 2 Jun 2011, at 03:32 , Steve Mallen wrote: > >> Hi all, >> >> I'm having problems processing a large lexicon of values and wondered if >> anyone had done something similar or had any ideas of how best to deal >> with them. >> >> Basically, I've got a set of several million distinct values, and I want >> to precompute a bunch of statistics for each of them (so that I can then >> facet/sort values on the computed statistic). So, my plan is to fetch >> all the values from the lexicon (storing them in a temp file, say), and >> then run an XQuery on each value and store the resulting information in >> a document (i.e. one stat document per value). I cannot do this in a >> single query as it would take far too long to iterate over all values >> and for all the computations and inserts. >> >> But I can't seem to figure out the best way of fetching and iterating >> over a Lexicon in MarkLogic (to pre-fetch the full set of lexicon >> values). In SQL, I'd use a CURSOR to fetch the values one by one, and >> then close the cursor at the end. There doesn't seem to be an analogous >> concept in XQuery or XCC. I've tried something along the following lines: >> >> (cts:element-values( xs:QName(lexi) ))[$start to $end] >> >> and fetching the values in blocks until I run out of values but I'm >> worried that this isn't very efficient, and I've got this nagging doubt >> that the above will never return the empty sequence when $start is past >> the end of the values. I'm not even sure how I should get a count of >> the number of distinct values (xdmp:estimate doesn't work on the result >> of cts:element-values()). >> >> So - do you guys know of a way of efficiently iterating over a large set >> of lexicon values without timing out the query on the server? >> >> If I'm missing an obvious solution, please let me know... >> >> -Steve >> >> _______________________________________________ >> General mailing list >> [email protected] >> http://developer.marklogic.com/mailman/listinfo/general >> > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
