Sini, There are ways to get the XQuery execution to not keep the documents in memory, but it is tricky and you still need to process all 50,000 within the transaction timeout. Spawning many smaller export tasks or using XQSync may be better.
Typically, consumers of data are better able to process a zip or tar archive of many XML documents, rather than a single huge XML document. Can you tell me more about what you need to accomplish? Yours, Damon From: [email protected] [mailto:[email protected]] On Behalf Of sini narayanan Sent: Tuesday, December 04, 2012 3:54 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Export from Marklogic Hi Damon, In this case, what if I have around 50000 ids in the list-of-IDs.xml file ? Would it cause a memory leak, since we are holding all the contents of the entire list of IDs in a varible? Will there be a workaround for this, something like appending the contents directly to the file created in the file system, without holding it in memory? Please share your thoughts. Thanks, Sini On Tue, Dec 4, 2012 at 9:24 AM, sini narayanan <[email protected]<mailto:[email protected]>> wrote: Hi Damon, Thanks for the help. The code is working fine. Thanks & Regards, Sini On Mon, Dec 3, 2012 at 5:53 PM, Harry B. <[email protected]<mailto:[email protected]>> wrote: Sini- You need to build your complete xml structure before the call to xdmp:save(). You are overwriting your previous file each time the script loops through. I think this is more what you're looking for (haven't tested it...check for errors): let $input-document := "/config/list-Of-IDs.xml" let $file-contents := for $data in fn:doc($input-document)/ids/id/text() let $fileName := fn:concat($data,".xml") return fn:doc($fileName)/node() let $compilation := <combinedData> {$file-contents} </combinedData> return xdmp:save("/tmp/Demo/example.xml", $compilation) This would give you the contents of each file from your config list inside the <combinedData> parent node in the example.xml file. If 1.xml had: <someNode> <foo>1</foo> </someNode> And 2.xml had <someNode> <foo>2</foo> </someNode> And only 1.xml and 2.xml were in your config list, example.xml would look like: <combinedData> <someNode> <foo>1</foo> </someNode> <someNode> <foo>2</foo> </someNode> </combinedData> On Mon, Dec 3, 2012 at 8:08 AM, sini narayanan <[email protected]<mailto:[email protected]>> wrote: Hi Damon, This will work when I explicitly know which are the files that i wnat to combine. But in my case, Im my Marklogic DB, I have 10 xml files named as 1.xml, 2.xml, .....10.xml And I have an xml file with list of IDs like below: <ids> <id>1</id> <id>2</id> <id>3</id> <id>4</id> <id>5</id> </ids> Now, For each of the ids in the list-Of-IDs.xml(1, 2 ,3, 4 and 5) i want to save the contents in the corresponding xml files into a single xml file in the file system. ie: Contents of 1.xml, 2.xml, 3.xml, 4.xml and 5.xml should be saved into a file named "example.xml" in the file system. when I iterate through the IDs in the xml like below, Only the final xml file contents is getting saved in the "example.xml" output file. Instead I want the contents of other id's as well to be appended/combined in the output. How will i do that ? Please help. let $input-document := "/config/list-Of-IDs.xml" for $data in doc($input-document)/ids/id let $ID := $data/text() let $fileName := concat($ID,".xml") return xdmp:save("/tmp/Demo/example.xml", doc($fileName)) Thanks On Mon, Dec 3, 2012 at 2:58 PM, Damon Feldman <[email protected]<mailto:[email protected]>> wrote: Sini, I'm not sure exactly what you mean, but you can do this with simple templating: let $doc1 := <doc><foo/></doc> let $doc2 := <doc2><foo2/></doc2> let $combined := <combined-doc>{$doc1/node(), $doc2/node()}</combined-doc> return xdmp:save("c:/tmp/combined-doc.xml", $combined) Yours, Damon -- Damon Feldman Sr. Principal Consultant, MarkLogic From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of sini narayanan Sent: Monday, December 03, 2012 8:39 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Export from Marklogic Hi All, Does anyone have an idea on how to combine two xml files and then save it in the file system using xdmp:save? Thanks On Mon, Dec 3, 2012 at 12:22 PM, sini narayanan <[email protected]<mailto:[email protected]>> wrote: Hi, I want to export the xml files inside Marklogic into a CSV in the file system. Is it possible to save a file inside repository into a file system using xquery ? Thanks _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
