What is the exact syntax do you use ? If your calling one ml:put for each file this can be a huge overhead as it needs to allocate a new connection for each file. I suspect the sockets are not getting garbage collected in time.
I suggest using the batch mode for xmlsh put which refuses the same XCC connection to send multiple files. I have successfully sent millions of files in one invocation with this. ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected] Phone: +1 650-287-2531 Cell: +1 812-630-7622 www.marklogic.com<http://www.marklogic.com/> This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. From: [email protected] [mailto:[email protected]] On Behalf Of Lanz Sent: Monday, June 04, 2012 6:22 PM To: general Subject: [MarkLogic Dev General] exception when puting in ML DB a large amout of data with XMLSH + ANT : "Cannot obtain connection: No buffer space available " Hi all, I'm trying to put a large amount (more than 28.000 files) of small files (around 5Kb each) into a Marklogic DB using XMLSH driven by ANT but when the process arrived at about the 4000th files I get this exception : [xmlsh] 2012-06-04 23:36:27.374 WARNING [10] (AbstractRequestController.runRequest): Cannot obtain connection: No buffer space available (maximum connections reached?): connect [xmlsh] java.net.SocketException: No buffer space available (maximum connections reached?): connect [xmlsh] sun.nio.ch.Net.connect(Native Method) [xmlsh] sun.nio.ch.SocketChannelImpl.connect(Unknown Source) [xmlsh] java.nio.channels.SocketChannel.open(Unknown Source) [xmlsh] com.marklogic.xcc.impl.SocketPoolProvider.obtainConnection(SocketPoolProvider.java:76) [xmlsh] com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:75) [xmlsh] com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:201) [xmlsh] com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:229) [xmlsh] org.xmlsh.marklogic.put.load(put.java:407) [xmlsh] org.xmlsh.marklogic.put.run(put.java:207) [xmlsh] org.xmlsh.core.XCommand.run(XCommand.java:106) [xmlsh] org.xmlsh.core.XCommand.run(XCommand.java:83) [xmlsh] org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:123) [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:691) [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:664) [xmlsh] org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:126) [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:691) [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:664) [xmlsh] org.xmlsh.sh.shell.Shell.runScript(Shell.java:436) [xmlsh] org.xmlsh.core.ScriptCommand.run(ScriptCommand.java:70) [xmlsh] org.xmlsh.commands.builtin.xmlsh.run(xmlsh.java:118) [xmlsh] org.xmlsh.core.BuiltinCommand.run(BuiltinCommand.java:54) [xmlsh] org.xmlsh.ant.XmlshTask.execute(XmlshTask.java:82) [xmlsh] org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) [xmlsh] org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) [xmlsh] org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62) [xmlsh] org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) [xmlsh] org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) [xmlsh] org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:394) [xmlsh] net.sf.antcontrib.logic.ForTask.doSequentialIteration(ForTask.java:259) [xmlsh] net.sf.antcontrib.logic.ForTask.doToken(ForTask.java:268) [xmlsh] net.sf.antcontrib.logic.ForTask.doTheTasks(ForTask.java:324) [xmlsh] net.sf.antcontrib.logic.ForTask.execute(ForTask.java:244) [xmlsh] org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) [xmlsh] org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) [xmlsh] org.apache.tools.ant.Target.execute(Target.java:357) [xmlsh] org.apache.tools.ant.Target.performTasks(Target.java:385) [xmlsh] org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337) [xmlsh] org.apache.tools.ant.Project.executeTarget(Project.java:1306) [xmlsh] org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) [xmlsh] org.apache.tools.ant.Project.executeTargets(Project.java:1189) [xmlsh] org.apache.tools.ant.Main.runBuild(Main.java:758) [xmlsh] org.apache.tools.ant.Main.startAnt(Main.java:217) [xmlsh] org.apache.tools.ant.launch.Launcher.run(Launcher.java:257) [xmlsh] org.apache.tools.ant.launch.Launcher.main(Launcher.java:104) [xmlsh] Exception submitting data I have tried to use " -XX:+UseConcMarkSweepGC" option but it did not change anything. For information I use XMLSH v1.1.9 (ML XCC Java 5.0-3 and latest extension module to xmlsh) on a Marklogic server standard edition 5.0.3). Thanks in advance. Lancelot
_______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
