I want to insert about 3 million 300 byteish docs to ML. I tried using RecordLoader and it did the trick but took about 10 hours.
Inserting the same document (fragmented to 300 mil fragments) as 1 document using XCC directly takes about 1 hour. Obviously things can be improved. Any suggestions on what might be fastest ? Suppose I have the 3 mil documents already split up in a directory on my local filesystem. After talking to ML Tech support it was suggested that by doing loads in batches would be faster then one at a time. Maybe I can do better then Record Loader. Any suggestions using XCC on which would be faster ? 1) ContentLoader.load( String[] , File[] ) 2) Session.insertContent( Content[] ) Another Idea I had was to split the 1G doc into say 1000 (instead of 3 mil) docs each containing 3000 elements then loading them into ML (unfragmented) then runningn an xquery program on the server to create the final 3mili documents. ---------------------------------------- David A. Lee Senior Principal Software Engineer Epocrates, Inc. [email protected] <mailto:[email protected]> 812-482-5224
_______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general
