Take a look at http://marklogic.markmail.org/thread/k56wrer44ufgaml5 for a more scalable technique in pure XQuery. It isn't immune to queue size limits, but it is easier to tune. Or use Corb: https://github.com/marklogic/corb.
-- Mike On 6 Jun 2011, at 20:19 , Tim Meagher wrote: > I use cts:uris to get a list of the uris that I want to delete and then I > spawn an xquery to delete each document. You have to be careful that you > don’t spawn more deletion tasks than can be accommodated on the task server > queue. > > Tim Meagher > > From: [email protected] > [mailto:[email protected]] On Behalf > [email protected] > Sent: Monday, June 06, 2011 10:07 PM > To: Marklogic Developer Mailing List > Subject: [MarkLogic Dev General] Deleting large number of documents from > large database > > I need to delete all of the documents in a collection in Marklogic. The > database in which we have the documents stored has over 4.4 million > documents. The collection itself has approximately 1.7 million documents. > We are currently running 4.0-4 of MarkLogic. We will be upgrading in the > near future to 4.2, but that does not help us with this problem. > > > I have attempted to use the code below to perform the delete, but I am only > able to delete 5 documents per second using this method. > > > > (: Delete documents in a collection :) > xquery version "1.0-ml"; > > let $assert := > if (xdmp:request-timestamp()) then () > else > error((), 'NOTIMESTAMP', text { 'outer query is not read-only' }) > let $path := '/documents/7615990080550346622/' > let $map := map:map() > let $list-uris := > for $i in xdmp:directory($path, 'infinity')[1 to 10000] > return map:put($map, xdmp:node-uri($i), true()) > let $do := xdmp:eval(' > declare variable $URIS as map:map external; > xdmp:document-delete(map:keys($URIS)) > ', > (xs:QName('URIS'), $map), > <options xmlns="xdmp:eval"> > <isolation>different-transaction</isolation> > <prevent-deadlocks>true</prevent-deadlocks> > </options> > ) > return count(map:keys($map)) > > I read another post on this mailing list that indicated that if you set > directory creation to "manual" and did not have any triggers enabled in the > database, that doing a directory delete would be extremely fast. However, we > DO have CPF enabled (and cannot disable it) and cannot set directory creation > to manual since we depend on the automatic directory creation in our > application. > > > > Any suggestions as to how we can get better throughput for deleting all of > the documents in a collection OR deleting a large number of documents from a > large database? Clearing a collection as quickly as possible while > retaining the collection itself is the goal at this time. > > > > Thanks, > > > > Allen Shatzer > > > > > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
