Hello, You could try using fastload if it would be applicable in your case.
If you're using an MLCP transform you may get better performance for ingestion by using a trigger instead. Of course, never use MLCP on a machine that's a MarkLogic host. Beyond that, the only other way I know of to scale out MLCP would be to break up the file into multiple smaller files across different machines and have individual instances of MLCP processing them. Best, Rob Rob Szkutak Associate Consultant MarkLogic Corporation [email protected] Cell +1.716.562.8464 www.marklogic.com<http://www.marklogic.com> ________________________________ From: [email protected] [[email protected]] on behalf of [email protected] [[email protected]] Sent: Saturday, September 19, 2015 5:06 PM To: [email protected] Subject: [MarkLogic Dev General] MLCP Data Migration Hi All, I am using MLCP to migrate CSV delimited file into MarkLogic (XML format). The file has 20 million record. Is there any way I can tune mlcp config to get optimized throughput. Current setting: THREAD=50 SPLIT_SIZE=20000 Current process is taking 1 - 1.5 Hrs to migrate 20MM Hardware configuration RAM: 128GB Cluster Size:1 Node Thanks Abhishek This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
_______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
