Hi Lanz. I'm sending you an email off list to help debug this. Thanks. ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected] Phone: +1 650-287-2531 Cell: +1 812-630-7622 www.marklogic.com<http://www.marklogic.com/>
This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. From: [email protected] [mailto:[email protected]] On Behalf Of Lanz Sent: Tuesday, June 05, 2012 9:25 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] exception when puting in ML DB a large amout of data with XMLSH + ANT : "Cannot obtain connection: No buffer space available " Hi again, I tried to use batch mode to put xml files from a directory into ML DB but I obtained only exceptions ! :-( I give context : 1- On a server 'S' there is a directory 'OUT' with several sub-directories (only one level) with several xml files inside 2- with ANT, I loop on these sub-directories with a 'dirset' task to call xmlsh to put their xml files into ML server here is the xmlsh code: XMODPATH=C:/Tools/xmlsh/lib/ import module ml=marklogic MLCONNECT="xcc://$1:$2@$3" ml:put -baseuri /content/ -maxfiles 10 OUT/$4 where $4 is a parameter given by ANT which stands for a sub-directory on 'S' server and which has to be created into ML server if it does not exist already. $1, $2 and $3 are the xcc connection parameters. 3- all I have as result is : "[xmlsh] Exception submitting data" Thanks in advance for any help Lancelot On Tue, Jun 5, 2012 at 12:42 AM, Lanz <[email protected]<mailto:[email protected]>> wrote: Hi David, you're right I'm calling one ml:put for each files, I will try with the batch mode for xmlsh put. Many thanks for your reactivity Lancelt On Tue, Jun 5, 2012 at 12:37 AM, David Lee <[email protected]<mailto:[email protected]>> wrote: What is the exact syntax do you use ? If your calling one ml:put for each file this can be a huge overhead as it needs to allocate a new connection for each file. I suspect the sockets are not getting garbage collected in time. I suggest using the batch mode for xmlsh put which refuses the same XCC connection to send multiple files. I have successfully sent millions of files in one invocation with this. ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected]<mailto:[email protected]> Phone: +1 650-287-2531<tel:%2B1%20650-287-2531> Cell: +1 812-630-7622<tel:%2B1%20812-630-7622> www.marklogic.com<http://www.marklogic.com/> This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Lanz Sent: Monday, June 04, 2012 6:22 PM To: general Subject: [MarkLogic Dev General] exception when puting in ML DB a large amout of data with XMLSH + ANT : "Cannot obtain connection: No buffer space available " Hi all, I'm trying to put a large amount (more than 28.000 files) of small files (around 5Kb each) into a Marklogic DB using XMLSH driven by ANT but when the process arrived at about the 4000th files I get this exception : [xmlsh] 2012-06-04 23:36:27.374 WARNING [10] (AbstractRequestController.runRequest): Cannot obtain connection: No buffer space available (maximum connections reached?): connect [xmlsh] java.net.SocketException: No buffer space available (maximum connections reached?): connect [xmlsh] sun.nio.ch.Net.connect(Native Method) [xmlsh] sun.nio.ch.SocketChannelImpl.connect(Unknown Source) [xmlsh] java.nio.channels.SocketChannel.open(Unknown Source) [xmlsh] com.marklogic.xcc.impl.SocketPoolProvider.obtainConnection(SocketPoolProvider.java:76) [xmlsh] com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:75) [xmlsh] com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:201) [xmlsh] com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:229) [xmlsh] org.xmlsh.marklogic.put.load(put.java:407) [xmlsh] org.xmlsh.marklogic.put.run(put.java:207) [xmlsh] org.xmlsh.core.XCommand.run(XCommand.java:106) [xmlsh] org.xmlsh.core.XCommand.run(XCommand.java:83) [xmlsh] org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:123) [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:691) [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:664) [xmlsh] org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:126) [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:691) [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:664) [xmlsh] org.xmlsh.sh.shell.Shell.runScript(Shell.java:436) [xmlsh] org.xmlsh.core.ScriptCommand.run(ScriptCommand.java:70) [xmlsh] org.xmlsh.commands.builtin.xmlsh.run(xmlsh.java:118) [xmlsh] org.xmlsh.core.BuiltinCommand.run(BuiltinCommand.java:54) [xmlsh] org.xmlsh.ant.XmlshTask.execute(XmlshTask.java:82) [xmlsh] org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) [xmlsh] org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) [xmlsh] org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62) [xmlsh] org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) [xmlsh] org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) [xmlsh] org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:394) [xmlsh] net.sf.antcontrib.logic.ForTask.doSequentialIteration(ForTask.java:259) [xmlsh] net.sf.antcontrib.logic.ForTask.doToken(ForTask.java:268) [xmlsh] net.sf.antcontrib.logic.ForTask.doTheTasks(ForTask.java:324) [xmlsh] net.sf.antcontrib.logic.ForTask.execute(ForTask.java:244) [xmlsh] org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) [xmlsh] org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) [xmlsh] org.apache.tools.ant.Target.execute(Target.java:357) [xmlsh] org.apache.tools.ant.Target.performTasks(Target.java:385) [xmlsh] org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337) [xmlsh] org.apache.tools.ant.Project.executeTarget(Project.java:1306) [xmlsh] org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) [xmlsh] org.apache.tools.ant.Project.executeTargets(Project.java:1189) [xmlsh] org.apache.tools.ant.Main.runBuild(Main.java:758) [xmlsh] org.apache.tools.ant.Main.startAnt(Main.java:217) [xmlsh] org.apache.tools.ant.launch.Launcher.run(Launcher.java:257) [xmlsh] org.apache.tools.ant.launch.Launcher.main(Launcher.java:104) [xmlsh] Exception submitting data I have tried to use " -XX:+UseConcMarkSweepGC" option but it did not change anything. For information I use XMLSH v1.1.9 (ML XCC Java 5.0-3 and latest extension module to xmlsh) on a Marklogic server standard edition 5.0.3). Thanks in advance. Lancelot _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://community.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
