You guys are the best!!!
It is unbelievable! The xdmp:spawn worked like a charm and it spit out
all the files in less than 15 minutes, whereas before the largest file
took an hour! The largest file is 663 megs.
Yeah! Thanks so very much!
Betty
> Hi Betty,
>
> You might start by creating a string range index on RGNum, then instead of
> the for loop you show, run a lexicon call to get the distinct values.
> Something like:
>
> cts:element-values(xs:QName("RGNum"), (), (),
> cts:and-query((
> cts:directory-query('/opt/documents/'),
> cts:element-query(xs:QName("era:MyDocument"), cts:and-query( () ))
> )) )
>
> That should make that distinct values calculation much faster.
>
> Now you could also xdmp:spawn each spreadsheet, passing the value from the
> distict-values calculation (or the element-values call if you create the
> range index).
>
> -Danny
>
> -----Original Message-----
> From: general-boun...@developer.marklogic.com
> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Betty Harvey
> Sent: Tuesday, February 14, 2012 6:06 PM
> To: MarkLogic Developer Discussion
> Subject: Re: [MarkLogic Dev General] How to Stop Time Out
>
> It is not possible to run the queries separately because I don't know what
> the unique values are. There are a possibility of 1100+ values but only
> 186 are currently in the database. Ultimately the number of documents are
> going to increase tremendously and all 1100 spreadsheets will need to be
> created. 2.4 million documents are the tip of the iceberg.
>
> I am getting the unique values:
>
> for $RG in
> fn:distinct-values(xdmp:directory('/opt/documents/')/descendant::era:MyDocument/descendant::RGNum/text())
>
> I then create the spreadsheet and call a function that gathers information
> for each document that matches the $RG.
>
> I have a variable that maintains the contents of each spreadsheet.
>
> The only command in the return is:
>
> xdmp:save($RGNum, document {$report} )
>
> This writes out every spreadsheet. I am running the report from the
> browser or CQ.
>
> Thanks!
>
> Betty
>
>> Hi Betty,
>>
>> You say 'the script stops at 36'. That sounds to me as if you are
>> creating
>> all 186 spreadsheets in one request. Is it possible to generate them one
>> by one? That might also allow parallel processing them.
>>
>> You also say that a larger case deals with a spreadsheet of 664k+ rows
>> and
>> takes an hour. You will have to tell more about the strategy you use in
>> order to help with optimizations. I have one thought though: could
>> taking
>> multiple steps be a solution?
>>
>> Colleen asks whether you are running this in the background or not. App
>> and Task server both have their own timeout setting. Background
>> processing
>> makes employing CPF interesting too.
>>
>> Kind regards,
>> Geert
>>
>>> -----Oorspronkelijk bericht-----
>>> Van: general-boun...@developer.marklogic.com [mailto:general-
>>> boun...@developer.marklogic.com] Namens Betty Harvey
>>> Verzonden: woensdag 15 februari 2012 0:04
>>> Aan: MarkLogic Developer Discussion
>>> Onderwerp: [MarkLogic Dev General] How to Stop Time Out
>>>
>>> Is there anyway to tell MarkLogic not to time out. I have developed a
>>> query that is cycles through the database and creates an Excel
>> spreadsheet
>>> for each distinct group of values.
>>>
>>> There are 2.4 million documents in the database. There should be a
>> total
>>> of 186 spreadsheets that get created. The script stops at 36
>>> spreadsheets. I tested one of the larger values that resulted in a
>> total
>>> of 664,000+ rows in the Excel spreadsheet and it took approximately an
>>> hour.
>>>
>>> I need someway of telling MarkLogic not to time out on this query. I
>>> really don't want to increase the time for timeout. I am working in a
>>> test environment and this option isn't an available option in
>> production.
>>>
>>> TIA!
>>>
>>> Betty
>>>
>>> /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
>>> Betty Harvey | Phone: 410-787-9200 FAX: 9830
>>> Electronic Commerce Connection, Inc. |
>>> har...@eccnet.com | Washington,DC XML Users Grp
>>> URL: http://www.eccnet.com | http://www.eccnet.com/xmlug
>>> /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\\/\/
>>> Member of XML Guild (www.xmlguild.org)
>>> _______________________________________________
>>> General mailing list
>>> General@developer.marklogic.com
>>> http://developer.marklogic.com/mailman/listinfo/general
>> _______________________________________________
>> General mailing list
>> General@developer.marklogic.com
>> http://developer.marklogic.com/mailman/listinfo/general
>>
>
>
> /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
> Betty Harvey | Phone: 410-787-9200 FAX: 9830
> Electronic Commerce Connection, Inc. |
> har...@eccnet.com | Washington,DC XML Users Grp
> URL: http://www.eccnet.com | http://www.eccnet.com/xmlug
> /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\\/\/
> Member of XML Guild (www.xmlguild.org)
> _______________________________________________
> General mailing list
> General@developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general
> _______________________________________________
> General mailing list
> General@developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general
>
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
Betty Harvey | Phone: 410-787-9200 FAX: 9830
Electronic Commerce Connection, Inc. |
har...@eccnet.com | Washington,DC XML Users Grp
URL: http://www.eccnet.com | http://www.eccnet.com/xmlug
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\\/\/
Member of XML Guild (www.xmlguild.org)
_______________________________________________
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
