At some scale, one simply cannot do everything in a single 'transaction' ( where 'transaction' refers to the database term). The semantics of a 'database transaction' require more then what is often really needed .. i.e. the 'common usage' of the word "transaction" often meaning "I can undo changes if things go bad and revert to the original." Maybe or maybe not affecting visibility by other 'transactions' while in progress.
One way that may be sufficient and can scale ... * Commit a list of URI's you are copying to a known URI - with a status field. * split the list into 'chunks' and spawn a task to do chunk[1] In the task * copy in chunks and update the lists in the same transaction as the 'chunk' copy * spawn another task to copy chunk[n+1] * To handle 'rollback' , delete already copied documents as indicated in the status field and update the field in the same transaction (or delete the whole list ) * To handle system shutdowns, at some common place in the code, such as a periodic task , check if there is a document at the 'copy URI' and if so, perform the 'rollback' or continue where you left off. -- There are many other ways to do this depending on what the observable behavior needs to be. Take a look at protected collections and document privileges -- you can 'hide' documents until they are ready to be 'seen' From: [email protected] [mailto:[email protected]] On Behalf Of Manoj Sent: Tuesday, May 10, 2016 11:03 AM To: MarkLogic Developer Discussion <[email protected]> Subject: Re: [MarkLogic Dev General] Need to copy more thatn 10k files in single transaction xdmp:spawn() will not support rollback/single transaction. If there is a failure I cannot rollback my changes done on some of the files which would have got updated in the DB. On Tue, May 10, 2016 at 4:43 AM, Asit Nautiyal <[email protected]<mailto:[email protected]>> wrote: Hi Manoj, Use xdmp:spawn() function. Regards, Asit Nautiyal ________________________________ From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> on behalf of Manoj <[email protected]<mailto:[email protected]>> Sent: Wednesday, May 4, 2016 7:07 PM To: General MarkLogic Developer Discussion Subject: [MarkLogic Dev General] Need to copy more thatn 10k files in single transaction Hi All, I need to copy more than 10k files from one directory to another after updating certain parts of the files in a single transaction(with ability to rollback in case of any failure). What is the best way to do it. When I was trying for larger set of file using xdmp:invoke with isolation as 'same-statement' I faced timeouts in marklogic Thanks & Regards Manoj _______________________________________________ General mailing list [email protected]<mailto:[email protected]> Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
