Wow ... sorry for the spam but I did it ! I got the 500 docs/sec. And no it wasnt the manual directory mode. it was a bug in my batching code (using XCC) , it was still sending 1 file at a time - darn typeo. I fixed it to send 500 files at a time and now I'm getting about 500 docs/sec !! Finally :)
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Lee, David Sent: Tuesday, December 08, 2009 9:21 PM To: General Mark Logic Developer Discussion Subject: RE: [MarkLogic Dev General] Inserting millions of small documents Jason, your a lifesaver !!! My directory delete was still running 3 hours later, I killed it, changed the directory create to manual, restarted the server and reran it (500k docs to go) 10 seconds later it was done ! Wow. Thank you ... I wonder if this manual create mode will help with creating the docs too ?? Michael suggests I should be getting 500 docs/sec created but I'm only getting about 20. I think I might try again with this setting to manual. Learning a lot every day, thanks everyone ! -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Jason Hunter Sent: Tuesday, December 08, 2009 7:51 PM To: General Mark Logic Developer Discussion Subject: Re: [MarkLogic Dev General] Inserting millions of small documents On Dec 8, 2009, at 3:41 PM, Lee, David wrote: > I've had just as bad experience doing a directory-delete as a for loop with document-delete ... Slooooooooowwwwww The fastest way to do a bulk delete is with a forest (or database) clear. Then it's close to instantaneous. The second fastest way is to do a collection-delete() or directory-delete(), but these can only be super fast if a few criteria are met. Here's the rules: -- When processing an xdmp:directory-delete(), the server chooses fast mode if: 1) The "directory creation" database setting is set to "manual" 2) There are no triggers set against the database being modified 3) No locks have been acquired in the database being modified (via xdmp:lock-aquire()) If all three conditions are met, then the server can simply use index resolution to identify all the fragments to "expire" and doesn't need to fetch any off disk in order to process locks, triggers, or enforce directory hierarchy to deal with orphaned documents/directories that may be left behind if the caller of xdmp:directory-delete() didn't have permissions to delete all the documents underneath "/foo/" (which can only happen under "manual" mode for directory creation). -- So set directory creation to manual, make sure you don't have CPF enabled when you do the call, and the delete should be very fast. -jh- _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general
