Hi Indy, You usually don’t need manual merging, MarkLogic will clean up automatically provided merging is not disabled. Deleted fragments might stick around for one hour, just to make sure merging does not slow down deletion itself..
Cheers, Geert From: <general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>> on behalf of Indrajeet Verma <indrajeet.ve...@gmail.com<mailto:indrajeet.ve...@gmail.com>> Reply-To: MarkLogic Developer Discussion <general@developer.marklogic.com<mailto:general@developer.marklogic.com>> Date: Monday, September 19, 2016 at 5:36 PM To: MarkLogic Developer Discussion <general@developer.marklogic.com<mailto:general@developer.marklogic.com>> Subject: Re: [MarkLogic Dev General] How does "xdmp:collection-delete" work? Hi Geert, In addition, do you think he should run manual merge after deleting 20M to free the space and improve performance? Regards, Indy On Mon, Sep 19, 2016 at 8:58 PM, Geert Josten <geert.jos...@marklogic.com<mailto:geert.jos...@marklogic.com>> wrote: Hi Qambar, I think it makes sense to discuss this in more detail here first, and then see if we can summarize conclusions on SO.. In general there are several ways to get rid of a large group of files. It generally comes down to either: 1. xdmp:collection-delete and xdmp:directory-delete 2. or a batch delete approach. This roughly matches the two answers on SO. The ‘benefit' of approach 1 is that it happens in one transaction, which could be important to you. But you are right that a collection-delete can take time. I would not necessarily say it will flood servers, but deleting 20 mln docs could take up to minutes. How much exactly depends a lot on factors like how many forests, how fast your disks are, how many MarkLogic instances you have in your cluster, how the docs are spread across those, etc. Deleting 20 mln docs could just as well take 10 sec, provided right configuration, and right circumstances are met. Right circumstances also includes things like not having triggers, not having enabled auditing etc.. The second approach has kind of the opposite. You won’t have the deletion happening in one transaction (unless you care to handle transactions yourself), but you have more control to manage load, and can take as long as needed. There are several tools that can help spawning deletion tasks. Corb/Corb2 is one, Taskbot is another. Which answer fits your case best, depends firstly on whether or not it is important to do the collection-delete in one transaction. Secondly, the volume of the average deletion counts, and how often you need to perform it. It might be good to run a test on a similar environment that allows estimating whether you can run the delete in an acceptable timeframe. We could go into more detail about xdmp:collection-delete, but I don’t think that will be of much help to you. Instead I’d prefer returning to your description on SO, you are talking about ‘expired’ collection items. Have you considered giving documents an expiry date, and running a schedule that will periodically remove expired documents? If the schedule runs for instance every hour, and would delete a reasonable sized batch of files on average, that could help spread load for keeping your system clean.. Cheers, Geert From: <general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>> on behalf of Qambar Raza <qambar.r...@bbc.co.uk<mailto:qambar.r...@bbc.co.uk>> Reply-To: MarkLogic Developer Discussion <general@developer.marklogic.com<mailto:general@developer.marklogic.com>> Date: Monday, September 19, 2016 at 1:00 PM To: "general@developer.marklogic.com<mailto:general@developer.marklogic.com>" <general@developer.marklogic.com<mailto:general@developer.marklogic.com>> Subject: [MarkLogic Dev General] How does "xdmp:collection-delete" work? Hello, Can anyone answer my question on stack overflow, I couldn't find a documentation about how https://docs.marklogic.com/xdmp:collection-delete works. For more details, see : http://stackoverflow.com/questions/39571215/how-does-marklogics-xdmpcollection-delete-work Thanks, Qambar. _______________________________________________ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
