Hi David, you're right I'm calling one ml:put for each files, I will try with the batch mode for xmlsh put. Many thanks for your reactivity
Lancelt On Tue, Jun 5, 2012 at 12:37 AM, David Lee <[email protected]> wrote: > What is the exact syntax do you use ?**** > > ** ** > > If your calling one ml:put for each file this can be a huge overhead as it > needs to allocate a new connection for each file. I suspect the sockets > are not getting garbage collected in time.**** > > ** ** > > I suggest using the batch mode for xmlsh put which refuses the same XCC > connection to send multiple files. I have successfully sent millions of > files in one invocation with this.**** > > ** ** > > ** ** > > > ----------------------------------------------------------------------------- > **** > > David Lee > Lead Engineer > MarkLogic Corporation > [email protected] > Phone: +1 650-287-2531 > Cell: +1 812-630-7622 > www.marklogic.com > > This e-mail and any accompanying attachments are confidential. The > information is intended solely for the use of the individual to whom it is > addressed. Any review, disclosure, copying, distribution, or use of this > e-mail communication by others is strictly prohibited. If you are not the > intended recipient, please notify us immediately by returning this message > to the sender and delete all copies. Thank you for your cooperation.**** > > ** ** > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Lanz > *Sent:* Monday, June 04, 2012 6:22 PM > *To:* general > *Subject:* [MarkLogic Dev General] exception when puting in ML DB a large > amout of data with XMLSH + ANT : "Cannot obtain connection: No buffer space > available "**** > > ** ** > > Hi all, > > I'm trying to put a large amount (more than 28.000 files) of small files > (around 5Kb each) into a Marklogic DB using XMLSH driven by ANT but when > the process arrived at about the 4000th files I get this exception : > [xmlsh] 2012-06-04 23:36:27.374 WARNING [10] > (AbstractRequestController.runRequest): Cannot obtain connection: No buffer > space available (maximum connections reached?): connect > [xmlsh] java.net.SocketException: No buffer space available (maximum > connections reached?): connect > [xmlsh] sun.nio.ch.Net.connect(Native Method) > [xmlsh] sun.nio.ch.SocketChannelImpl.connect(Unknown Source) > [xmlsh] java.nio.channels.SocketChannel.open(Unknown Source) > [xmlsh] > com.marklogic.xcc.impl.SocketPoolProvider.obtainConnection(SocketPoolProvider.java:76) > [xmlsh] > com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:75) > [xmlsh] > com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:201) > [xmlsh] > com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:229) > [xmlsh] org.xmlsh.marklogic.put.load(put.java:407) > [xmlsh] org.xmlsh.marklogic.put.run(put.java:207) > [xmlsh] org.xmlsh.core.XCommand.run(XCommand.java:106) > [xmlsh] org.xmlsh.core.XCommand.run(XCommand.java:83) > [xmlsh] > org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:123) > [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:691) > [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:664) > [xmlsh] org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:126) > [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:691) > [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:664) > [xmlsh] org.xmlsh.sh.shell.Shell.runScript(Shell.java:436) > [xmlsh] org.xmlsh.core.ScriptCommand.run(ScriptCommand.java:70) > [xmlsh] org.xmlsh.commands.builtin.xmlsh.run(xmlsh.java:118) > [xmlsh] org.xmlsh.core.BuiltinCommand.run(BuiltinCommand.java:54) > [xmlsh] org.xmlsh.ant.XmlshTask.execute(XmlshTask.java:82) > [xmlsh] > org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) > [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) > [xmlsh] > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) > [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) > [xmlsh] > org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62) > [xmlsh] > org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) > [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) > [xmlsh] > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) > [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) > [xmlsh] > org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:394) > [xmlsh] > net.sf.antcontrib.logic.ForTask.doSequentialIteration(ForTask.java:259) > [xmlsh] net.sf.antcontrib.logic.ForTask.doToken(ForTask.java:268) > [xmlsh] > net.sf.antcontrib.logic.ForTask.doTheTasks(ForTask.java:324) > [xmlsh] net.sf.antcontrib.logic.ForTask.execute(ForTask.java:244) > [xmlsh] > org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) > [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) > [xmlsh] > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) > [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) > [xmlsh] org.apache.tools.ant.Target.execute(Target.java:357) > [xmlsh] org.apache.tools.ant.Target.performTasks(Target.java:385) > [xmlsh] > org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337) > [xmlsh] > org.apache.tools.ant.Project.executeTarget(Project.java:1306) > [xmlsh] > org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) > [xmlsh] > org.apache.tools.ant.Project.executeTargets(Project.java:1189) > [xmlsh] org.apache.tools.ant.Main.runBuild(Main.java:758) > [xmlsh] org.apache.tools.ant.Main.startAnt(Main.java:217) > [xmlsh] org.apache.tools.ant.launch.Launcher.run(Launcher.java:257) > [xmlsh] > org.apache.tools.ant.launch.Launcher.main(Launcher.java:104) > [xmlsh] Exception submitting data > > I have tried to use " -XX:+UseConcMarkSweepGC" option but it did not > change anything. > For information I use XMLSH v1.1.9 (ML XCC Java 5.0-3 and latest extension > module to xmlsh) on a Marklogic server standard edition 5.0.3). > Thanks in advance. > Lancelot**** > > _______________________________________________ > General mailing list > [email protected] > http://community.marklogic.com/mailman/listinfo/general > >
_______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
