Hi again, I tried to use batch mode to put xml files from a directory into ML DB but I obtained only exceptions ! :-( I give context : 1- On a server 'S' there is a directory 'OUT' with several sub-directories (only one level) with several xml files inside 2- with ANT, I loop on these sub-directories with a 'dirset' task to call xmlsh to put their xml files into ML server here is the xmlsh code:
XMODPATH=C:/Tools/xmlsh/lib/ import module ml=marklogic MLCONNECT="xcc://$1:$2@$3" ml:put -baseuri /content/ -maxfiles 10 OUT/$4 where $4 is a parameter given by ANT which stands for a sub-directory on 'S' server and which has to be created into ML server if it does not exist already. $1, $2 and $3 are the xcc connection parameters. 3- all I have as result is : "[xmlsh] Exception submitting data" Thanks in advance for any help Lancelot On Tue, Jun 5, 2012 at 12:42 AM, Lanz <[email protected]> wrote: > Hi David, > > you're right I'm calling one ml:put for each files, I will try with the > batch mode for xmlsh put. > Many thanks for your reactivity > > Lancelt > > On Tue, Jun 5, 2012 at 12:37 AM, David Lee <[email protected]>wrote: > >> What is the exact syntax do you use ?**** >> >> ** ** >> >> If your calling one ml:put for each file this can be a huge overhead as >> it needs to allocate a new connection for each file. I suspect the sockets >> are not getting garbage collected in time.**** >> >> ** ** >> >> I suggest using the batch mode for xmlsh put which refuses the same XCC >> connection to send multiple files. I have successfully sent millions of >> files in one invocation with this.**** >> >> ** ** >> >> ** ** >> >> >> ----------------------------------------------------------------------------- >> **** >> >> David Lee >> Lead Engineer >> MarkLogic Corporation >> [email protected] >> Phone: +1 650-287-2531 >> Cell: +1 812-630-7622 >> www.marklogic.com >> >> This e-mail and any accompanying attachments are confidential. The >> information is intended solely for the use of the individual to whom it is >> addressed. Any review, disclosure, copying, distribution, or use of this >> e-mail communication by others is strictly prohibited. If you are not the >> intended recipient, please notify us immediately by returning this message >> to the sender and delete all copies. Thank you for your cooperation.**** >> >> ** ** >> >> *From:* [email protected] [mailto: >> [email protected]] *On Behalf Of *Lanz >> *Sent:* Monday, June 04, 2012 6:22 PM >> *To:* general >> *Subject:* [MarkLogic Dev General] exception when puting in ML DB a >> large amout of data with XMLSH + ANT : "Cannot obtain connection: No buffer >> space available "**** >> >> ** ** >> >> Hi all, >> >> I'm trying to put a large amount (more than 28.000 files) of small files >> (around 5Kb each) into a Marklogic DB using XMLSH driven by ANT but when >> the process arrived at about the 4000th files I get this exception : >> [xmlsh] 2012-06-04 23:36:27.374 WARNING [10] >> (AbstractRequestController.runRequest): Cannot obtain connection: No buffer >> space available (maximum connections reached?): connect >> [xmlsh] java.net.SocketException: No buffer space available (maximum >> connections reached?): connect >> [xmlsh] sun.nio.ch.Net.connect(Native Method) >> [xmlsh] sun.nio.ch.SocketChannelImpl.connect(Unknown Source) >> [xmlsh] java.nio.channels.SocketChannel.open(Unknown Source) >> [xmlsh] >> com.marklogic.xcc.impl.SocketPoolProvider.obtainConnection(SocketPoolProvider.java:76) >> [xmlsh] >> com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:75) >> [xmlsh] >> com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:201) >> [xmlsh] >> com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:229) >> [xmlsh] org.xmlsh.marklogic.put.load(put.java:407) >> [xmlsh] org.xmlsh.marklogic.put.run(put.java:207) >> [xmlsh] org.xmlsh.core.XCommand.run(XCommand.java:106) >> [xmlsh] org.xmlsh.core.XCommand.run(XCommand.java:83) >> [xmlsh] >> org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:123) >> [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:691) >> [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:664) >> [xmlsh] org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:126) >> [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:691) >> [xmlsh] org.xmlsh.sh.shell.Shell.exec(Shell.java:664) >> [xmlsh] org.xmlsh.sh.shell.Shell.runScript(Shell.java:436) >> [xmlsh] org.xmlsh.core.ScriptCommand.run(ScriptCommand.java:70) >> [xmlsh] org.xmlsh.commands.builtin.xmlsh.run(xmlsh.java:118) >> [xmlsh] org.xmlsh.core.BuiltinCommand.run(BuiltinCommand.java:54) >> [xmlsh] org.xmlsh.ant.XmlshTask.execute(XmlshTask.java:82) >> [xmlsh] >> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) >> [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown >> Source) >> [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> Source) >> [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) >> [xmlsh] >> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) >> [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) >> [xmlsh] >> org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62) >> [xmlsh] >> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) >> [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown >> Source) >> [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> Source) >> [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) >> [xmlsh] >> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) >> [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) >> [xmlsh] >> org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:394) >> [xmlsh] >> net.sf.antcontrib.logic.ForTask.doSequentialIteration(ForTask.java:259) >> [xmlsh] net.sf.antcontrib.logic.ForTask.doToken(ForTask.java:268) >> [xmlsh] >> net.sf.antcontrib.logic.ForTask.doTheTasks(ForTask.java:324) >> [xmlsh] net.sf.antcontrib.logic.ForTask.execute(ForTask.java:244) >> [xmlsh] >> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) >> [xmlsh] sun.reflect.GeneratedMethodAccessor3.invoke(Unknown >> Source) >> [xmlsh] sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> Source) >> [xmlsh] java.lang.reflect.Method.invoke(Unknown Source) >> [xmlsh] >> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) >> [xmlsh] org.apache.tools.ant.Task.perform(Task.java:348) >> [xmlsh] org.apache.tools.ant.Target.execute(Target.java:357) >> [xmlsh] org.apache.tools.ant.Target.performTasks(Target.java:385) >> [xmlsh] >> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337) >> [xmlsh] >> org.apache.tools.ant.Project.executeTarget(Project.java:1306) >> [xmlsh] >> org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) >> [xmlsh] >> org.apache.tools.ant.Project.executeTargets(Project.java:1189) >> [xmlsh] org.apache.tools.ant.Main.runBuild(Main.java:758) >> [xmlsh] org.apache.tools.ant.Main.startAnt(Main.java:217) >> [xmlsh] >> org.apache.tools.ant.launch.Launcher.run(Launcher.java:257) >> [xmlsh] >> org.apache.tools.ant.launch.Launcher.main(Launcher.java:104) >> [xmlsh] Exception submitting data >> >> I have tried to use " -XX:+UseConcMarkSweepGC" option but it did not >> change anything. >> For information I use XMLSH v1.1.9 (ML XCC Java 5.0-3 and latest >> extension module to xmlsh) on a Marklogic server standard edition 5.0.3). >> Thanks in advance. >> Lancelot**** >> >> _______________________________________________ >> General mailing list >> [email protected] >> http://community.marklogic.com/mailman/listinfo/general >> >> >
_______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
