Hello Wayne,
I think you want to move the line:
using (var connection = Use.Connection(ConnectionStringSettings))
so it's outside the foreach loop. If nothing else, that should improve the
performance so a new connection isn't being opened for each row processed.
-Bill
On Tuesday, 7 May 2013 05:42:18 UTC-4, Wayne Douglas wrote:
> Hi
>
> Thanks for that, firstly, what is an MVCC timeout/
>
> Also - I have now broken the data out by club but how can I feed that club
> id into the next input operation? Convention input operation doesn't have
> the ability to pass params to the sproc that is executed to fetch the data?
>
> I have created my own input operation which can do this but it is slow as
> hell. It accepts a single param as that is all I need for now. Here is the
> code: http://monobin.com/1568/
>
> w
>
> On Sunday, 5 May 2013 16:15:57 UTC+1, Nathan Palmer wrote:
>>
>> Depends. Since it's timing out mid way through processing I assume
>> manipulating the command and/or connection timeout won't help you. I assume
>> you are running into a MVCC timeout (depends on backend database or
>> configuration.) The best thing might be to page based on content. Find a
>> good way to split the data and create an operation that runs first to grab
>> the distinct values. Then when you get to the main select you filter on
>> that value. To give an example if we're dealing with addresses you can have
>> this.
>>
>> operation GetDistinctStates
>> operation GetData
>> operation ProcessData
>>
>> GetDistinctStates will get a list of states within your dataset
>> GetData will process 1 state at a time (and yield it to the next
>> operation)
>> ProcessData will do what it's doing today.
>>
>> Nathan Palmer
>>
>>
>> On Sun, May 5, 2013 at 7:42 AM, Wayne Douglas <wa...@prjatk.com> wrote:
>>
>>> I'm getting timeout issues when doing a select on a huge data set - it
>>> times out in the select mid way through processing.
>>>
>>> Is there anyway to page through huge data sets to stop this from
>>> happening?
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Rhino Tools Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rhino-tools-dev+unsubscr...@googlegroups.com.
To post to this group, send email to rhino-tools-dev@googlegroups.com.
Visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
