And for cases where mlcp doesn't quite do the job yet, http://marklogic.github.io/recordloader/ might.
But if documents are taking one second each, batching is unlikely to help: transaction overhead simply isn't that significant. Batching is mostly helpful for cases where there are many, very short transactions. Getting back to the problem, my first question is, "why is it taking an entire second per document?" How large are the documents? What sort of processing is taking place, both in terms of indexing and other enrichment or management activities? Which subsystem appears to be the bottleneck? -- Mike On 3 Jun 2013, at 09:14 , David Lee <[email protected]> wrote: > What code are you using for ingestion ? > I suggest looking at mlcp which does not only batch but parallel (threaded) > ingestion. > > http://developer.marklogic.com/products/mlcp > > ----------------------------------------------------------------------------- > David Lee > Lead Engineer > MarkLogic Corporation > [email protected] > Phone: +1 812-482-5224 > Cell: +1 812-630-7622 > www.marklogic.com > > > From: [email protected] > [mailto:[email protected]] On Behalf Of Bob O > Sent: Monday, June 03, 2013 11:57 AM > To: [email protected] > Subject: [MarkLogic Dev General] Time of ingestion > > Ladies/Gentlemen, > > I was wondering if there is a way to batch process documents being ingested > as oppose of ingesting it one at a time. Our documents are taking one > document per second and it looks slow to me. I was thinking maybe ingesting > 500 docs as a single file at a time to speed up the ingestion. > > any ideas? > > > ~~BobO > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
