Damon,
I am not sure what you mean by "1 per transaction in your eval". Are you
referring to the "different-transaction" isolation? If so, my understanding,
based on the MarkLogic documentation is that this simply runs the query
(delete)
in a different transaction from the calling transaction. I would assume since
we are using a "map" to store 10,000 URIs that those 10000 URIs would be
deleted
in a SINGLE transaction, not each in their own transaction.
Also, isn't this code URI-based? (In response to your comment "Certainly moving
to something URIs based will help avoid the initial transaction loading the
full
documents."). If not, what is the code getting when it it calls
"xdmp:node-uri($i))"?
If this is not the case, how would you re-write this to batch the delete? I
got this code from a another post on this mailing list and it appears to be
batching the deletes, but I am an xquery novice.
Regarding the CPF, we are using MarkLogic's replication to keep a remote
MarkLogic database in sync for disaster recovery and failover purposes. It is
not feasible to disable the CPF or change the way it works.
In response to other poster's responses, it is not possible to change the
design
of the database / application at this time to address this issue since the
application is in production use.
As I noted above, I am an xquery novice, so any assistance to re-work the code
below OR another method that would be more efficient would be greatly
appreciated.
Thanks,
Allen
________________________________
From: Damon Feldman <[email protected]>
To: General MarkLogic Developer Discussion <[email protected]>
Sent: Tue, June 7, 2011 7:20:31 AM
Subject: Re: [MarkLogic Dev General] Deleting large number of documents from
large database
Allen,
Certainly moving to something URIs based will help avoid the initial
transaction
loading the full documents. Batching the deletes may also help (100 or 1000 per
transaction, rather than 1 per transaction in your eval()).
The CPF processing may be the culprit. Does CPF run on this domain, and if so
is
it working hard to figure out whether to run or not when these documents are
deleted, or doing actual work? Monitoring your task server for excessive
activity can shed light on the task server load. Also logging start and
end-times for your processes if you have that in place.
Yours,
Damon
From:[email protected]
[mailto:[email protected]] On Behalf Of
[email protected]
Sent: Monday, June 06, 2011 10:07 PM
To: Marklogic Developer Mailing List
Subject: [MarkLogic Dev General] Deleting large number of documents from large
database
I need to delete all of the documents in a collection in Marklogic. The
database in which we have the documents stored has over 4.4 million
documents. The collection itself has approximately 1.7 million documents.
We
are currently running 4.0-4 of MarkLogic. We will be upgrading in the near
future to 4.2, but that does not help us with this problem.
I have attempted to use the code below to perform the delete, but I am only
able
to delete 5 documents per second using this method.
(: Delete documents in a collection :)
xquery version "1.0-ml";
let $assert :=
if (xdmp:request-timestamp()) then ()
else
error((), 'NOTIMESTAMP', text { 'outer query is not read-only' })
let $path := '/documents/7615990080550346622/'
let $map := map:map()
let $list-uris :=
for $i in xdmp:directory($path, 'infinity')[1 to 10000]
return map:put($map, xdmp:node-uri($i), true())
let $do := xdmp:eval('
declare variable $URIS as map:map external;
xdmp:document-delete(map:keys($URIS))
',
(xs:QName('URIS'), $map),
<options xmlns="xdmp:eval">
<isolation>different-transaction</isolation>
<prevent-deadlocks>true</prevent-deadlocks>
</options>
)
return count(map:keys($map))
I read another post on this mailing list that indicated that if you set
directory creation to "manual" and did not have any triggers enabled in the
database, that doing a directory delete would be extremely fast. However, we
DO
have CPF enabled (and cannot disable it) and cannot set directory creation to
manual since we depend on the automatic directory creation in our application.
Any suggestions as to how we can get better throughput for deleting all of the
documents in a collection OR deleting a large number of documents from a large
database? Clearing a collection as quickly as possible while retaining the
collection itself is the goal at this time.
Thanks,
Allen Shatzer_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
